PDF analysis method and system based on image-text semantic alignment

Abstract

The invention provides a PDF analysis method and system based on image-text semantic alignment, and belongs to the technical field of document processing. The method comprises the following steps: reading PDF or picture file byte data, and converting pictures into PDF byte data in a unified format; creating an image and a Markdown directory according to the output directory, the PDF name and an analysis method; extracting byte data of a specified page of the PDF according to starting and ending page numbers; selecting a back end to analyze PDF byte data to obtain a reasoning result, an image list and the like; generating an intermediate JSON containing PDF detailed information according to an analysis result; analyzing the positions of the text and the picture, and associating information to realize semantic alignment of the picture and the text; and generating multiple types of machine readable files according to the intermediate JSON and storing the files in a specified directory. According to the method, the PDF multi-mode content can be accurately extracted and subjected to semantic alignment, the PDF multi-mode content is efficiently converted into a machine readable format, the analysis accuracy and usability are improved, and the method is suitable for PDF analysis of multiple image-text tables such as scientific and technical literatures.

Description

Technical Field

[0001] This invention belongs to the field of document processing technology, specifically relating to a PDF parsing method and system based on image-text semantic alignment. Background Technology

[0002] Vision and language, as two important mediums for information representation, storage, and dissemination, play crucial roles in daily production and life. In recent years, with the development of computer technology and artificial intelligence, the demand for machine perception and reasoning has been increasing, especially for the understanding and matching of image and text data. Image-text retrieval, as a fundamental and critical task in information retrieval and multimedia computing, can not only effectively bridge the semantic gap and distributional barriers between vision and language, but also promote the development of many higher-level applications, such as cross-modal retrieval, image annotation, and visual question answering, thus possessing broad and profound research significance.

[0003] PDF documents, due to their cross-platform compatibility and ability to retain the original layout, have become an important medium for information dissemination and storage. As a widely used electronic document format, PDF documents often contain both text and image information, making PDF parsing and image retrieval technologies closely related.

[0004] However, traditional PDF parsing methods often process text and images separately, failing to fully consider the semantic relationships between them. This results in parsing results that cannot accurately reflect the overall semantic information of the document, making it difficult to meet the needs for in-depth understanding and efficient utilization of document content. Especially in the field of scientific and technological literature, PDF documents contain numerous images, tables, formulas, and other elements, and these elements have complex semantic relationships with the text. Traditional PDF parsing methods struggle to accurately understand and process this complex semantic information, leading to inaccurate parsing results that fail to meet the needs of practical applications.

[0005] Meanwhile, existing PDF parsing technologies primarily focus on text extraction and layout analysis, exhibiting weak capabilities in handling semantic alignment between text and images. They fail to effectively address the semantic gap between text and image modalities, layout complexity, and dynamic adaptability issues. While some methods can identify images and tables, they cannot accurately establish the semantic relationship between them and the text, resulting in the loss of crucial contextual information in the parsed document. Furthermore, existing parsing methods struggle to handle complex scientific documents, such as those containing chemical formulas, geometric figures, or other special elements. Therefore, developing a PDF parsing method capable of achieving semantic alignment between text and images is of significant practical importance. Summary of the Invention

[0006] This invention provides a PDF parsing method and system based on image-text semantic alignment to solve the technical problems existing in the prior art, such as weak image-text semantic alignment capabilities in multi-focus text extraction and layout analysis, difficulty in bridging the image-text semantic gap, layout complexity, and handling of complex elements in scientific and technological documents, resulting in inaccurate parsing results, loss of context, and inability to meet the needs of in-depth document understanding and practical applications.

[0007] To achieve the above objectives, the present invention adopts the following technical solution: A PDF parsing method based on image-text semantic alignment includes the following steps: Read the file to be parsed, perform multimodal file parsing, and output a structured PDF; Generate a multi-level storage structure based on the output structure requirements; Get the page number specified by the user, analyze the continuity of edge elements of the structured PDF page, automatically merge cross-page content blocks through context semantic association algorithm, and extract page data within the specified page number range; Based on the extracted page data, the physical coordinates and visual attributes of different elements in the page data are captured, and a logical relationship network between elements is established. Based on the physical coordinates, visual attributes and logical relationship network, the document is reconstructed into a tree model. Multimodal analysis is performed on the tree model document to output structured content. Based on structured content, an image-text association index is automatically built through a rule engine. Based on the image-text association index, an image-text alignment is performed through a three-layer progressive multi-dimensional alignment strategy to build an intermediate JSON dataset. Based on the intermediate JSON dataset, multi-format standardized output and version management are performed, and the document content of the tree model is reconstructed. The reconstructed file is saved according to the preset path of the multi-level storage structure.

[0008] The multimodal file parsing process involves: reading the byte data of the file to be parsed; for image files, using a multimodal conversion engine to unify the format; firstly, optimizing low-quality scanned documents using image enhancement algorithms; then, using computer vision technology to locate text regions and visual elements; combining PaddleOCR V5 to extract text content; and finally generating a structured PDF byte stream according to the original visual layout, ensuring that information such as text, charts, and symbols in the image is completely preserved.

[0009] The process of generating a multi-level storage structure based on output structure requirements involves automatically generating a three-level storage structure based on the user-specified output root directory, PDF source file name, and parsing method. The three-level storage structure consists of a main folder prefixed with the PDF file name, and internal subdirectories named "images," "markdown," and "temp." "Images" stores extracted image resources according to the "page number_element ID" rule; "markdown" is used to save the structured output results; and "temp" caches intermediate data such as paginated PDFs and OCR temporary files.

[0010] The process of extracting page data within a specified page range is based on the PyPDFium2 engine, which extracts page data within a specified page range from the PDF byte data. Page number specification supports both absolute and relative page number input, and the page number validity is automatically verified during processing. The page data includes metadata such as the original byte stream, size, and rotation angle, and supports splitting and storing it as a temporary PDF by single page.

[0011] The process involves extracting page data, capturing the physical coordinates and visual attributes of different elements within the page data, and establishing logical relationships between these elements. Based on these logical relationships, the document structure is organized into a tree model. Specifically, a visual detection engine based on Grounding DINO accurately captures the physical coordinates and visual attributes of different elements in the page data, including font specifications, color coding, spacing matrices, and relative positioning relationships. Simultaneously, a logical relationship network between elements is established through a semantic understanding module. Based on the element attributes in this logical relationship network, the hierarchical reconstruction of the document structure is performed using a detection model finely tuned with LayoutLMv4 to detect regions, identify the macro-structure of the document, and organize the article structure into a tree model. This records the structural information of each part of the article, such as the title and abstract, preparing for semantic alignment of text and images and the output of structured content.

[0012] Multimodal analysis is performed on the tree model document to output structured content. Specifically: tables are reconstructed using graph neural networks, formulas are located by YOLOv11 and then converted to LaTeX format by UniMERNet V2, and images are first generated as visual feature vectors by CLIP model and then as preliminary descriptive text by LLaVA-Next, finally outputting structured content.

[0013] Based on structured content, a rule engine automatically builds an image-text association index: Regular expressions are used to match citation markers in the document text, including "Fig.X" and "Fig.X" which point to specific visual elements. The element numbers identified in the markers are extracted, and the position coordinates of each visual element on the document page are obtained. Spatial correlation is determined between the position coordinates of the text block containing the citation marker and the coordinates of the corresponding visual element, generating a preliminary mapping. For content without explicit citations, semantic similarity is used to calculate the associated context text blocks, constructing a standardized middle_json dataset file. The standardized middle_json dataset file includes a three-layer structure: document metadata, page-level data, and element-level data.

[0014] Based on the image-text association index, an image-text alignment strategy is adopted to construct an intermediate JSON dataset. Specifically: at the basic spatial alignment level, the vertical distance and horizontal overlap rate between the text block and the image are calculated, and a context window of 300 characters before and after the image is dynamically extracted. An adaptive sliding window algorithm is used to intelligently identify paragraph boundaries and avoid semantic breaks caused by abrupt character truncation, thus constructing an initial association pool. At the semantic alignment level, key entity recognition is performed on the extracted context text, and cross-modal attention calculation is performed with the visual feature vector generated by the CLIP model to generate a joint embedding representation. At the logical alignment level, the logical relationship between the image and text is analyzed through GPT-4V, and a "relation_type" field is added to the intermediate JSON to record the deep association between text and image.

[0015] Based on the intermediate JSON dataset, multi-format standardized output and version management are implemented, and the document content is reconstructed according to the tree model. Specifically, the structured Markdown file is formatted according to the generated tree model, with text retaining heading levels, tables converted to Markdown format, and images inserted in the format: . Formulas are embedded with LaTeX code. The content list JSON is a simplified version of the data, retaining text, image paths, and core related information, suitable for fast retrieval scenarios. Based on the OCR recognition results of the multimodal document analysis module and the data captured by the visual inspection engine, quality indicators such as OCR confidence and layout recognition score are recorded for result traceability and quality assessment. All output files are saved according to the multi-level storage structure specified by the user in the cross-system adapted storage environment construction module.

[0016] A PDF parsing system based on image-text semantic alignment includes a multimodal file parsing module, a cross-system adaptable storage environment construction module, a page processing and semantic repair module, a multimodal document analysis module, a semantically related intermediate JSON construction module, and a tree model document reconstruction module. The multimodal file parsing module is used to read the file to be parsed, perform multimodal file parsing, and output a structured PDF; The cross-system adaptable storage environment construction module is used to generate a multi-level storage structure according to the output structure requirements; The page processing and semantic repair module is used to obtain the page number specified by the user, analyze the continuity of edge elements of the structured PDF page, automatically merge cross-page content blocks through the context semantic association algorithm, and extract page data within the specified page number range. The multimodal document analysis module is used to capture the physical coordinates and visual attributes of different elements in the extracted page data, establish a logical relationship network between elements, reconstruct the document into a tree model based on the physical coordinates, visual attributes and logical relationship network, perform multimodal analysis on the tree model document, and output structured content. The semantically related intermediate JSON building module is used to automatically build an image-text association index based on structured content through a rule engine, and to perform image-text alignment based on the image-text association index through a three-layer progressive multi-dimensional alignment strategy to build an intermediate JSON dataset. The document reconstruction module of the tree model is used to perform multi-format standardized output and version management based on the intermediate JSON dataset, and to reconstruct the document according to the content of the tree model document. The reconstructed file is saved according to the preset path of the multi-level storage structure.

[0017] Compared with the prior art, the present invention has the following beneficial effects: This invention proposes a PDF parsing method based on image-text semantic alignment. Utilizing image enhancement algorithms, computer vision technology, and PaddleOCR V5's multimodal file parsing capabilities, it performs unified format processing on image files: first, it optimizes the quality of low-quality scanned documents; then, it accurately locates text regions and visual elements; and finally, it extracts the text content and generates a structured PDF byte stream according to the original visual layout. This method can completely preserve information such as text, charts, and symbols in images, ensuring the integrity and accuracy of multimodal input data and providing a high-quality data foundation for subsequent document parsing stages.

[0018] Furthermore, this invention utilizes a three-level storage structure generated based on user-specified parameters. The main folder is prefixed with the PDF filename and contains subdirectories for images, markdown, and temp. Images are stored according to the page number-element ID rule, enabling standardized management of parsed resources. Image resources support breakpoint resumption and reuse. The markdown directory centrally stores structured output results, and the temp directory caches intermediate data. This structure improves the orderliness of file storage, reduces redundant data processing, and provides a clear path for subsequent file retrieval and retrieval, thereby improving overall processing efficiency.

[0019] Furthermore, page processing techniques employing the PyPDFium2 engine and contextual semantic association algorithms can accurately extract page data within a specified page range, automatically verify page number validity to avoid erroneous operations, and analyze the continuity of page edge elements and merge cross-page content blocks. This approach ensures accurate acquisition of target page data, corrects issues such as table breaks and paragraph incoherence caused by page breaks, guarantees the integrity and coherence of page content, and prevents the loss of critical information due to page segmentation.

[0020] Furthermore, document analysis based on the Grounding DINO visual inspection engine and semantic understanding module can accurately capture the physical coordinates and visual attributes of page elements and establish a logical relationship network between elements. Combined with the LayoutLMv4 fine-tuning model to construct a document tree model, it can identify the macro-structure of the document and record information such as titles and summaries. This method can clearly clarify the logical relationships between document elements, completely restore the original PDF's layout logic, provide structural support for subsequent content reconstruction, and ensure that the output file's layout is consistent with the original document.

[0021] Furthermore, specialized processing mechanisms are implemented for different elements, providing precise handling for complex elements such as tables, formulas, and images. Specifically, cross-page or nested tables can be effectively reconstructed, formulas can be converted to standard LaTeX format, and images can generate semantic descriptions. This approach meets the parsing needs of complex elements in professional documents such as scientific literature, ensuring the professionalism and accuracy of the parsing results for various elements.

[0022] Furthermore, the intermediate JSON dataset constructed in this invention comprises a three-layer structure: document metadata, page-level data, and element-level data. Combined with a rule engine to establish a text-image association index, it can integrate full-dimensional document information and automatically achieve initial text-image mapping and semantic association. This method provides a standardized data carrier with a clear data structure containing complete text-image association information. It supports subsequent in-depth three-layer progressive semantic alignment and provides a unified data interface for secondary development and third-party system integration, improving data reusability and system compatibility.

[0023] Furthermore, by calculating distance and overlap rate through basic spatial alignment and extracting contextual windows, key entity recognition and cross-modal attention calculation through semantic alignment, and textual logical relationship analysis through logical alignment using the GPT-4V model, the degree of association between text and images can be gradually deepened based on the above three-layer progressive image-text semantic alignment strategy. This method can clarify the logical relationships such as explanations and examples between images and text, automatically preserve contextual referential relationships, avoid semantic breaks, and ultimately construct a complete image-text semantic association system, significantly improving the semantic completeness and depth of document parsing.

[0024] Furthermore, this invention also encompasses multiple output formats, including structured Markdown and simplified JSON content lists. Combined with quality control measures that record OCR confidence and layout recognition scores, it can adapt to different usage scenarios: structured Markdown meets the needs of detailed reading and display, while simplified JSON is suitable for fast retrieval scenarios; quality indicators provide traceable evaluation criteria for the parsing results. This approach not only expands the application scope of the parsing results but also ensures the reliability of the results through quality control, thereby enhancing practical application value. Attached Figure Description

[0025] Figure 1 This is a framework diagram of the document multimodal parsing toolchain in an embodiment of the present invention; Figure 2 This is a schematic diagram of the module used to parse PDF in an embodiment of the present invention; Figure 3 This is a flowchart of a PDF parsing system based on image-text semantic alignment in an embodiment of the present invention; Figure 4 This is a flowchart of the image-text semantic alignment implementation part in an embodiment of the present invention. Detailed Implementation

[0026] To further understand the content of this invention, the invention will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the embodiments are merely illustrative and not limiting of the invention.

[0027] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0028] This invention provides a PDF parsing method based on image-text semantic alignment. Through multimodal content extraction, semantic alignment mechanisms, and standardized output, it achieves efficient parsing of PDF documents. It supports extracting text, images, tables, and other elements from PDF files and establishing semantic relationships, ultimately outputting in machine-readable formats such as Markdown and JSON. The PDF parsing process includes the following steps: Read the file to be parsed, perform multimodal file parsing, and output a structured PDF; Generate a multi-level storage structure based on the output structure requirements; Get the page number specified by the user, analyze the continuity of edge elements of the structured PDF page, automatically merge cross-page content blocks through context semantic association algorithm, and extract page data within the specified page number range; Based on the extracted page data, the physical coordinates and visual attributes of different elements in the page data are captured, and a logical relationship network between elements is established. Based on the physical coordinates, visual attributes and logical relationship network, the document is reconstructed into a tree model. Multimodal analysis is performed on the tree model document to output structured content. Based on structured content, an image-text association index is automatically built through a rule engine. Based on the image-text association index, an image-text alignment is performed through a three-layer progressive multi-dimensional alignment strategy to build an intermediate JSON dataset. Based on the intermediate JSON dataset, multi-format standardized output and version management are performed, and the document content of the tree model is reconstructed. The reconstructed file is saved according to the preset path of the multi-level storage structure.

[0029] Based on the above steps, combined with Figure 2 The flowchart shown in this embodiment is explained in detail as follows: S1: This embodiment supports importing documents from multiple sources, including local paths, network links, and cloud storage. Cloud storage options include Alibaba Cloud OSS and AWS S3. It also supports single-file and batch imports, with a maximum of 1000 files processed in parallel at a time. For PDF files, byte data is directly extracted. For image files such as JPG, PNG, and TIFF, a self-developed multimodal conversion engine is used to achieve format unification: first, adaptive denoising and perspective correction are performed using image enhancement algorithms to optimize low-quality scanned documents; then, computer vision technology is used to locate text regions and visual elements in the image; combined with OCR technology, text content is extracted; and finally, a structured PDF byte stream is generated according to the original visual layout, converting the image into PDF byte data and ensuring the complete preservation of text, charts, symbols, and other information in the image, ultimately outputting a structured PDF. Simultaneously, file metadata, including name, creation time, page number, and hash value, is automatically recorded, providing a foundation for subsequent version management and duplicate file detection.

[0030] S2: Based on the user-specified start / end page numbers, the target page byte data is extracted using a PyPDFium2-based pagination engine. The continuity of page edge elements is analyzed simultaneously, including chart splitting across pages and table page breaks. Cross-page content blocks are automatically merged using a contextual semantic association algorithm. In this embodiment, the user-specified page number input supports both absolute and relative page numbers, such as "Page 3 to the end". During the environment preparation phase, a multi-level storage structure is generated according to the path "output directory / filename / [images|markdown|temp]". In this embodiment, a three-level storage structure is generated, including an images directory, a markdown directory, and a temp directory. The images directory stores extracted image resources according to the "page number_element ID" rule. The markdown directory is used to save structured output results. The temp directory caches intermediate data such as paginated PDFs and OCR temporary files, and automatically configures cross-operating system directory permissions, such as adapting to Windows' NTFS permissions and Linux's chmod permissions. An operation log is also generated, including creation time and processing status, for problem tracing. After completing the environment preparation, page processing is performed. Then, based on the start / end page numbers specified by the user, the target page byte data is extracted using a pagination engine based on PyPDFium2.

[0031] S3: Employs an upgraded document multimodal parsing toolchain, performing a global layout scan of PDF files based on the Grounding DINO model. This accurately identifies the position and attributes of elements such as titles, body text, images, tables, formulas, footnotes, and sidenotes, including coordinates, font size, and color. It also merges paragraphs of multi-column text according to semantic logic. The document multimodal parsing toolchain includes a tree model intelligent construction module, a multimodal element classification module, a table intelligent reconstruction module, a formula deep analysis module, and an image content pre-parsing module. Specifically, the tree model intelligent construction module innovatively reconstructs the hierarchical structure of the document, organizing the article structure into a tree model and recording the structural information of each part, such as the title and abstract, preparing for semantic alignment of text and images and structured output. The table intelligent reconstruction module employs dedicated processing mechanisms for different element types: tables are reconstructed using graph neural networks (GNNs), supporting nested table parsing and cross-page merging while preserving row and column relationships and cell merging status; the formula deep parsing module uses the YOLOv11 model to locate regions, combined with UniMERNetV2 to convert to LaTeX format, and also supports complex symbols such as chemical structural formulas and matrix formulas; the multimodal element classification module and image content pre-parsing module, during image recognition, first call the CLIP model to generate visual feature vectors, and then use the LLaVA-Next multimodal large model to generate preliminary descriptive text; simultaneously, for the recognition of special symbols, non-text elements such as circuit diagram symbols and flowchart arrows are classified and labeled. All element information is integrated according to the page dimension to form a structured JSON file, providing structured data for subsequent processing.

[0032] S4: Based on the structured JSON file output from the parsing, a `middle_json` structure is constructed to generate a standardized `middle_json` dataset. The `middle_json` structure comprises three layers: document metadata, page data, and element details. The document metadata layer includes filenames, total number of pages, processing time, and engine version. The page data layer includes a list of elements per page, dimensions, and rotation angle. The element details layer includes text block content and coordinates, table cell data, image paths and feature vectors, and LaTeX formula code. During the construction of the standardized `middle_json` dataset, a rule engine automatically establishes an image-text association index: regular expressions are used to match image references in the text, combined with element position coordinates to generate an initial mapping; for content without explicit references, semantic similarity is used to calculate and associate related text blocks in the context, ensuring that each element is bound to a corresponding semantic description, providing a data foundation for subsequent alignment. This format is compatible with mainstream secondary development scenarios, such as knowledge base construction and data mining, and supports seamless integration with third-party systems such as Notion and Confluence.

[0033] S5: During the intermediate JSON generation process, image-text semantic alignment is achieved through a two-dimensional approach: spatial and semantic. Spatially, the vertical distance and horizontal overlap rate between the text block and the image are calculated. In this embodiment, the vertical distance between the images is <50 pixels, and the horizontal overlap rate is >30%. The physical location association is determined by calculating the vertical distance and horizontal overlap rate between the text block and the image. Semantically, a multimodal model is invoked, and a structured description is generated from the input image and adjacent text blocks. The structured description includes the topic, key elements, and contextual logical relationships. Then, a semantic similarity algorithm is used to compare the descriptive text with the associated text blocks to strengthen the semantic binding. The alignment results are stored in the form of an "image-text association matrix," recording the association confidence of each text block with the image. The parsed text of the image is embedded in the "semantic_annotation" field of the intermediate JSON, forming a two-layer alignment structure of "location association + semantic enhancement," which solves the problem of shallow association caused by the reliance on position or reference tags in traditional methods.

[0034] S6: Generates diverse outputs based on intermediate JSON data: Structured Markdown files are formatted according to the tree model of the articles generated in S3, ensuring that the generated text has the same formatting as the original PDF. Tables are converted to Markdown tables (supporting merged cell markers). Images are inserted in the format of [Image X Description](path "parse: [detailed description of multimodal generation]"). Formulas are embedded with LaTeX code. The content list JSON (simplified version) retains only the text, image paths, and core related information, suitable for fast retrieval. The model output file records quality indicators such as OCR confidence and layout recognition score. All files are stored according to standardized paths and a README.txt (containing a file list, processing time, and quality rating) is generated, supporting one-click package download or push to third-party systems (such as enterprise CMS) via API. For complex elements (such as charts and formulas), progressive rendering technology is used, outputting basic content first and then asynchronously loading high-definition rendering results to improve user experience.

[0035] S7: Identifies updated versions of processed PDFs by comparing file hash values, re-parseing only newly added and modified pages, reducing redundant calculations and improving efficiency by over 30%. During the verification phase, a visual verification interface is provided through table structures and formula conversion effects, allowing users to view the parsing results and manually adjust the relationship between images and text using interactive tools, such as correcting incorrectly associated images and text blocks. The adjustments are synchronized to the intermediate JSON in real time. Simultaneously, a parsing quality report is generated, marking low-confidence elements, such as text with an OCR recognition score <0.8 or areas with blurred layout recognition, providing users with optimization suggestions, such as re-uploading high-resolution files, forming a closed-loop processing mechanism of "automatic parsing + manual verification".

[0036] Furthermore, in step S3, a document multimodal parsing toolchain is used to achieve accurate parsing of all elements. This toolchain consists of five core modules, such as... Figure 1 As shown, the modules work together to form a complete chain of "global layout → local extraction → semantic enhancement", as detailed below: The intelligent block parsing module performs global layout detection based on the LayoutLMv4 model. It uses multi-scale feature fusion technology to locate the coordinates and attributes (such as font size and line spacing) of elements like titles, body text, images, tables, and footnotes. For complex structures such as multi-column layouts and nested text boxes (such as sidebars in technical documents), it employs an attention mechanism to prioritize semantically coherent areas, ensuring paragraph merging conforms to reading logic and resolving the problem of chaotic multi-column text splitting in traditional models.

[0037] Multimodal element classification module: Combining the visual-text cross-modal features of the CLIP model, the detected elements are finely classified: images are divided into subtypes such as "illustration", "data chart" and "photo", tables are divided into "data table", "comparison table" and "nested table", and formulas are divided into "inline formula", "independent formula block" and "chemical structure", providing a basis for subsequent special processing.

[0038] The table intelligent reconstruction module uses a graph neural network (GNN) to model the structure of table regions. It identifies merged cells and breaks in cross-page tables by analyzing the topological relationships between nodes (cells) and edges (row-column relationships), and repairs table integrity by combining contextual semantics. For example, for experimental data tables spanning two pages, it automatically associates the headers and data rows of the preceding and following pages, generating structured data containing "row index → ​​column index → ​​cell content," which can be directly exported to Excel or CSV format. The repair accuracy is 40% higher than traditional methods.

[0039] The formula deep analysis module uses the YOLOv11 model to locate formula regions, calls the UniMERNet V2 model for symbol recognition, converts printed / handwritten formulas into LaTeX code, and annotates the formula semantics using a mathematical symbol knowledge base (e.g., automatically adding the explanation "E=mc²" when recognizing "E=mc²"). For chemical structural formulas, it supports the recognition of special symbols such as benzene rings and functional groups, generating editable ChemDraw compatible formats.

[0040] Image content pre-parsing module: The extracted images are preprocessed (edges are cropped and contrast is enhanced), and then preliminary descriptive text is generated through the LLaVA-Next multimodal model (such as "Figure 3 is a PDF parsing system architecture diagram, which includes three core modules: data reading, document analysis, and semantic alignment"). Key entities in the images (such as "module name" and "arrow pointing relationship") are extracted and stored as feature vectors, providing a foundation for subsequent semantic alignment.

[0041] Furthermore, in step S5, a three-layer alignment mechanism of "spatial anchoring → semantic verification → logical enhancement" is constructed: Spatial association anchoring: Calculate the spatial distance between text blocks and images based on the page coordinate system: For elements with a vertical distance of <50 pixels and a horizontal overlap rate of >30%, they are automatically marked as "potential associations" and prioritize association with the title text above the image (such as "Figure 2 Schematic diagram of experimental device") and the explanatory text below it (such as "Note: A in the figure is the reaction vessel") to form basic association pairs.

[0042] Semantic similarity verification: Sentence-BERT is used to semantically encode the pre-parsed text of the image (image content pre-parsed from S3) and related text blocks, and cosine similarity is calculated: if the similarity is ≥0.7, it is determined to be "strongly associated" and the binding is strengthened; if the similarity is <0.5, the spatial association is removed to avoid incorrect binding (such as the image being mistakenly associated with an unrelated adjacent text block). For example, the text block containing "Figure 2 shows the model accuracy" is compared with the image parsing text "Figure 2 is a bar chart comparing the accuracy of different models". If the similarity meets the standard, a strong association is established.

[0043] Enhanced Logical Relationships: The GPT-4V model deeply analyzes the logical relationships between text and images, automatically identifying association types such as "explanation," "example," and "comparison" (e.g., "The text describes the experimental steps, and the image is a schematic diagram of the corresponding steps," which belongs to the "explanation relationship"), and adding a "relation_type" field to the intermediate JSON to record this. Simultaneously, detailed parsing text generated by the multimodal model (e.g., "The horizontal axis in the graph represents the training epochs, the vertical axis represents accuracy, and the curve shows that the model converged at epoch 50") is embedded into the associated text blocks, forming a triangular support structure of "text → image → parsing," thus solving the semantic gap problem caused by relying solely on positional associations in traditional methods.

[0044] like Figure 3 As shown, this embodiment also proposes a PDF parsing system based on image-text semantic alignment, including a multimodal file parsing module, a cross-system adaptable storage environment construction module, a page processing and semantic repair module, a multimodal document analysis module, an intermediate JSON construction module, a three-layer progressive image-text semantic alignment module, and a tree model reconstruction module; Multimodal file parsing module: This module uses a self-developed file processing engine to read the byte data of uploaded PDF or image files, including JPG, PNG, and TIFF formats. For PDF files, it directly extracts the byte data; for image files, a multimodal conversion engine performs format unification: first, it optimizes low-quality scanned documents using image enhancement algorithms that include adaptive denoising and perspective correction; then, it uses computer vision technology to locate text regions and visual elements, and combines this with PaddleOCR V5 to extract the text content. Finally, it generates a structured PDF byte stream according to the original visual layout, ensuring the complete preservation of text, charts, symbols, and other information in the image. It also integrates a file verification mechanism, automatically identifying corrupted files and encrypted PDFs and returning detailed error messages, such as "File encrypted: Password protection needs to be removed," ensuring the reliability of subsequent processing.

[0045] A cross-system compatible storage environment building module: Based on the user-specified output root directory, PDF source file name, and parsing method (e.g., OCR priority, layout recognition priority), a three-level storage structure is automatically generated: the main folder is prefixed with the PDF file name, and internally divided into "images", "markdown", and "temp" subdirectories. "images" stores extracted image resources according to the "page number_element ID" rule, supporting breakpoint resumption and resource reuse; "markdown" is used to save structured output results; and "temp" caches intermediate data such as paginated PDFs and OCR temporary files. The system automatically configures cross-operating system directory permissions, such as adapting to Windows' NTFS permissions and Linux's chmod permissions, and generates log files containing creation time and processing status for easy problem tracing and process auditing.

[0046] Page processing and semantic repair module: Based on the PyPDFium2 engine, this module extracts page data within a specified page range from the PDF byte data. This embodiment supports absolute and relative page number input, such as absolute page numbers 1-5 and relative page numbers from page 3 to the end. During processing, the module automatically verifies page number validity to prevent the starting page from being larger than the ending page or exceeding the total number of pages. During extraction, the module simultaneously analyzes the continuity of page edge elements, such as text spanning multiple pages or table splits. It automatically merges cross-page content blocks using a contextual semantic association algorithm, including repairing broken table rows and continuous paragraph text. The page data contains metadata such as the original byte stream, size, and rotation angle. It supports splitting and storing the data as a temporary PDF on a single page basis, facilitating individual OCR or image extraction for specific pages and improving processing flexibility.

[0047] The multimodal document analysis module, based on a tree model, performs differentiated analysis according to the user-selected backend engine: the backend engine can be either pipeline or VLM. If the pipeline mode is selected, the system employs a hybrid analysis method at the underlying processing level: a visual detection engine based on Grounding DINO accurately captures the physical coordinates and visual attributes of page elements, including but not limited to font specifications, color coding, spacing matrices, and relative positioning relationships; simultaneously, a semantic understanding module establishes a logical network of relationships between elements. This dual analysis mechanism ensures that each text paragraph, image block, table cell, and formula component is assigned precise geometric descriptors and semantic labels. For hierarchical reconstruction of the document structure, the system innovatively uses a detection model fine-tuned from LayoutLMv4 for region detection, organizing the article structure into a tree model and recording the structural information of each part of the article, such as the title and abstract, preparing for semantic alignment of text and images and structured output. This algorithm comprehensively considers multi-dimensional features such as visual spacing, layout mode, and font gradation rules to automatically infer the hierarchical topology of the document. Furthermore, dedicated processing mechanisms are enabled for different elements in the page data. The table structure is reconstructed through a graph neural network, supporting cross-page merging and nested table parsing. Formulas are located by YOLOv11 and then converted to LaTeX format by UniMERNet V2. Images are first processed by the CLIP model to generate visual feature vectors, and then LLaVA-Next is used to generate preliminary descriptive text to achieve document analysis.

[0048] If the backend engine uses the VLM mode, it directly calls the GPT-4V multimodal large model to complete page understanding in one go and output structured content. The analysis results include inferred text and coordinates, image lists grouped by page (including size and location), multilingual tags (such as language partitions for Chinese-English mixed pages), and OCR enabled tags (marking pages without native text). Parallel processing is supported to improve efficiency.

[0049] The semantically related intermediate JSON building module constructs a standardized `middle_json` dataset based on document analysis results. This dataset comprises three layers: document metadata, page-level data, and element-level data. Document metadata includes filenames, total number of pages, processing time, and engine version. Page-level data includes text blocks, tables, image references, and page dimensions for each page. Element-level data includes text block content and coordinates, table cell structure, image storage path, and confidence level. During the intermediate JSON generation process, a rule engine automatically establishes a text-image association index: Regular expressions are used to match reference markers in the document text, including "Fig.X" and "Fig.X" which point to specific visual elements. The element numbers identified in these markers are extracted, and the position coordinates of each visual element on the document page are obtained. Spatial correlation is determined between the position coordinates of the text block containing the reference marker and the coordinates of the corresponding visual element, generating a preliminary mapping. For content without explicit references, semantic similarity is used to calculate the associated context text blocks, ensuring that each element is bound to a corresponding semantic description. The constructed intermediate JSON format is compatible with the needs of mainstream secondary development scenarios, including knowledge base construction and data mining based on the JSON format of this embodiment. It also supports seamless integration with third-party systems, including Notion and Confluence related documentation tools. Without complicated additional adaptations, the data in the JSON of this embodiment can be used directly, or the JSON = data can be exchanged with the data of these systems.

[0050] A three-layer progressive multi-dimensional alignment strategy is used to achieve deep association between text and images, such as... Figure 4 As shown. At the basic spatial alignment level, not only are the vertical distance and horizontal overlap rate between the text block and the image calculated, but also the context window of 300 characters before and after the image is dynamically extracted, automatically skipping irrelevant elements such as headers and footers, and intelligently identifying paragraph boundaries through an adaptive sliding window algorithm to avoid semantic breaks caused by abrupt character truncation, thus constructing an initial association pool; the threshold for determining the vertical distance between the text block and the image is <50 pixels, and the threshold for determining the horizontal overlap rate between the text block and the image is >30%; if the vertical distance is <50 pixels and the horizontal overlap rate is >30%, then semantic relevance is determined and the text is bound; otherwise, it is marked as uncertain. This process uses an adaptive sliding window algorithm to intelligently identify paragraph boundaries and avoid semantic breaks caused by abrupt character truncation; at the semantic alignment level, key entity recognition is performed on the extracted context text, including key entities such as names of people, places, and technical terms, and cross-modal attention calculation is performed with the image feature vector generated by CLIP to generate a joint embedding representation.

[0051] Key entity recognition is performed on the contextual text extracted from the basic spatial alignment layer, covering names of people, places, technical terms, etc., and cross-modal attention is calculated with the image feature vectors generated by CLIP to generate a joint embedding representation. Then, Sentence-BERT is called for semantic encoding, and cosine similarity is calculated, with classification based on the similarity results. When the cosine similarity is ≥0.7, it is marked as "strongly related" and the context fusion process is triggered—when the image description text is inserted into Markdown in the format "parse: [description]", the referential relationships in the original context are automatically preserved, ensuring a natural connection between the inserted content and the surrounding narrative. Cosine similarity Pairings are marked as potentially related. If the cosine similarity is less than 0.5, the pairing is removed. In addition to removing the pairing, the semantic conflict type is also recorded, such as "the image shows a building while the text discusses plants", for reference in subsequent manual review.

[0052] For cases where semantic relevance is determined by spatial analysis and the content is bound together, or where semantic relevance is determined to be strong or potentially related, the GPT-4V model is used at the logical alignment level to deeply analyze the logical relationship between the text and image content, identifying types such as "explanation," "example," and "comparison." For instance, "the text describes the experimental steps, and the image is an operational diagram" belongs to the "explanation relationship." A "relation_type" field is added to the intermediate JSON to record this relationship, and detailed parsing text is generated to form a triangular support structure of "text → image → parsing." Finally, the JSON is updated, and the alignment result is stored as a "text-image association matrix," recording the association confidence of each text block with the image, thus solving the information fragmentation problem caused by text-image separation.

[0053] The tree-model document reconstruction module achieves standardized multi-format output and version management of documents based on intermediate JSON data. It reconstructs the article content using the tree model generated by the multimodal document analysis module, ensuring the reconstructed file maintains the same layout as the original PDF. Structured Markdown files are formatted according to the generated tree model, retaining heading levels in the text, converting tables to Markdown format (including merged cell markers), and inserting images in the format: . Formulas are embedded with LaTeX code. The content list JSON is a simplified version, retaining text, image paths, and core related information, suitable for rapid retrieval scenarios. Based on the OCR recognition results from the multimodal document analysis module and data captured by the visual inspection engine, it records quality indicators such as OCR confidence and layout recognition score for result traceability and quality assessment. All output files follow a three-level storage structure specified by the user in the storage environment building module, saved according to the path "output directory / filename / [markdown|images|logs]". A README.txt file is generated synchronously, containing a file list, processing time, and quality rating, forming a complete output archive. This archive supports one-click download or push to third-party systems via API. For complex elements such as tables and formulas, progressive rendering technology is employed. Basic content is output first to ensure quick preview for users, then high-definition rendered results are loaded asynchronously, further enhancing the user experience. This ultimately completes the entire closed-loop process from PDF upload and parsing to multi-format deliverables.

[0054] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can be appropriately combined to form other embodiments that can be understood by those skilled in the art. The above content is only for illustrating the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. Any modifications made based on the technical concept proposed in this invention shall fall within the scope of protection of the claims of this invention.

Claims

1. A PDF parsing method based on image-text semantic alignment, characterized in that, Includes the following steps: Read the file to be parsed, perform multimodal file parsing, and output a structured PDF; Generate a multi-level storage structure based on the output structure requirements; Get the page number specified by the user, analyze the continuity of edge elements of the structured PDF page, automatically merge cross-page content blocks through context semantic association algorithm, and extract page data within the specified page number range; Based on the extracted page data, the physical coordinates and visual attributes of different elements in the page data are captured, and a logical relationship network between elements is established. Based on the physical coordinates, visual attributes and logical relationship network, the document is reconstructed into a tree model. Multimodal analysis is performed on the tree model document to output structured content. Based on structured content, an image-text association index is automatically built through a rule engine. Based on the image-text association index, an image-text alignment is performed through a three-layer progressive multi-dimensional alignment strategy to build an intermediate JSON dataset. Based on the intermediate JSON dataset, multi-format standardized output and version management are performed, and the document content of the tree model is reconstructed. The reconstructed file is saved according to the preset path of the multi-level storage structure.

2. The PDF parsing method based on image-text semantic alignment according to claim 1, characterized in that, The process of multimodal file parsing involves: reading the byte data of the file to be parsed; for image files, using a multimodal conversion engine to unify the format; firstly, optimizing low-quality scanned documents using image enhancement algorithms; then, using computer vision technology to locate text regions and visual elements; combining PaddleOCR V5 to extract text content; and finally generating a structured PDF byte stream according to the original visual layout, ensuring that the text, charts, and symbol information in the image are completely preserved.

3. The PDF parsing method based on image-text semantic alignment according to claim 1, characterized in that, The process of generating a multi-level storage structure based on output structure requirements involves automatically generating a three-level storage structure based on the user-specified output root directory, PDF source file name, and parsing method. The three-level storage structure consists of a main folder prefixed with the PDF file name, and internal subdirectories for images, markdown, and temp. The "images" folder stores extracted image resources according to the "page number_element ID" rule, the "markdown" folder is used to save the structured output results, and the "temp" folder caches intermediate data, including paginated PDFs and OCR temporary files.

4. The PDF parsing method based on image-text semantic alignment according to claim 1, characterized in that, The extraction of page data within a specified page range is based on the PyPDFium2 engine, which extracts page data within a specified page range from the PDF byte data. Page number specification supports both absolute and relative page number input, and the page number validity is automatically verified during processing. The page data contains metadata about the original byte stream, size, and rotation angle, and supports splitting and storing it as a temporary PDF by single page.

5. A PDF parsing method based on image-text semantic alignment according to claim 1, characterized in that, The process of organizing the document structure into a tree model involves: using a visual inspection engine based on Grounding DINO to accurately capture the physical coordinates and visual attributes of different elements in the page data, including font specifications, color coding, spacing matrices, and relative positioning relationships; simultaneously, establishing a logical relationship network between elements through a semantic understanding module; and based on the element attributes in the logical relationship network, performing region detection using a fine-tuned LayoutLMv4 detection model to identify the macro-structure of the document and organize the article structure into a tree model, recording the structural information of each part of the article, such as the title and abstract, to prepare for semantic alignment of text and images and the output of structured content.

6. A PDF parsing method based on image-text semantic alignment according to claim 5, characterized in that, Multimodal analysis is performed on the tree model document to output structured content. Specifically: tables are reconstructed using graph neural networks, formulas are located by YOLOv11 and then converted to LaTeX format by UniMERNet V2, and images are first generated as visual feature vectors by CLIP model and then as preliminary descriptive text by LLaVA-Next, finally outputting structured content.

7. A PDF parsing method based on image-text semantic alignment according to claim 6, characterized in that, Based on structured content, a rule engine automatically builds an image-text association index: Regular expressions are used to match citation markers in the document text, including "Fig.X" and "Fig.X" which point to specific visual elements. The element numbers identified in the markers are extracted, and the position coordinates of each visual element on the document page are obtained. Spatial correlation is determined between the position coordinates of the text block containing the citation marker and the coordinates of the corresponding visual element, generating a preliminary mapping. For content without explicit citations, semantic similarity is used to calculate the associated context text blocks, constructing a standardized middle_json dataset file. The standardized middle_json dataset file includes a three-layer structure: document metadata, page-level data, and element-level data.

8. A PDF parsing method based on image-text semantic alignment according to claim 7, characterized in that, Based on the image-text association index, an image-text alignment is performed through a three-layer progressive multi-dimensional alignment strategy to construct an intermediate JSON dataset. Specifically, at the basic spatial alignment level, the vertical distance and horizontal overlap rate between the text block and the image are calculated, and the context window of 300 characters before and after the image is dynamically extracted. An adaptive sliding window algorithm is used to intelligently identify paragraph boundaries, avoid semantic breaks caused by abrupt character truncation, and construct an initial association pool. At the semantic alignment level, key entity recognition is performed on the extracted context text, and cross-modal attention calculation is performed on the visual feature vector generated by the CLIP model to generate a joint embedding representation; at the logical alignment level, the logical relationship between the text and the image is analyzed by GPT-4V, and a "relation_type" field is added to the intermediate JSON to record the deep association between the text and the image.

9. A PDF parsing method based on image-text semantic alignment according to claim 8, characterized in that, Based on the intermediate JSON dataset, multi-format standardized output and version management are performed, and the document content is reconstructed according to the tree model. Specifically, the structured Markdown file is formatted according to the generated tree model, retaining the text heading hierarchy, tables are converted to Markdown format, and images are inserted in the format: . Formulas are embedded with LaTeX code. The content list JSON is a simplified version of the data, retaining text, image paths, and core related information, suitable for fast retrieval scenarios. Based on the OCR recognition results of the multimodal document analysis module and the data captured by the visual inspection engine, quality indicators, including OCR confidence and layout recognition score, are recorded for result traceability and quality assessment. All output files are saved according to the file paths of the multi-level storage structure specified by the user in the storage environment building module.

10. A PDF parsing system based on image-text semantic alignment, comprising a PDF parsing method based on image-text semantic alignment as described in any one of claims 1 to 9, characterized in that, It includes a multimodal file parsing module, a cross-system adaptable storage environment construction module, a page processing and semantic repair module, a multimodal document analysis module, a semantically related intermediate JSON construction module, and a tree model document reconstruction module; The multimodal file parsing module is used to read the file to be parsed, perform multimodal file parsing, and output a structured PDF; The cross-system adaptable storage environment construction module is used to generate a multi-level storage structure according to the output structure requirements; The page processing and semantic repair module is used to obtain the page number specified by the user, analyze the continuity of edge elements of the structured PDF page, automatically merge cross-page content blocks through the context semantic association algorithm, and extract page data within the specified page number range. The multimodal document analysis module is used to capture the physical coordinates and visual attributes of different elements in the extracted page data, establish a logical relationship network between elements, reconstruct the document into a tree model based on the physical coordinates, visual attributes and logical relationship network, perform multimodal analysis on the tree model document, and output structured content. The semantically related intermediate JSON building module is used to automatically build an image-text association index based on structured content through a rule engine, and to perform image-text alignment based on the image-text association index through a three-layer progressive multi-dimensional alignment strategy to build an intermediate JSON dataset. The document reconstruction module of the tree model is used to perform multi-format standardized output and version management based on the intermediate JSON dataset, and to reconstruct the document according to the content of the tree model document. The reconstructed file is saved according to the preset path of the multi-level storage structure.

Images