PDF document intelligent word segmentation method based on content structure perception

Abstract

The invention provides a PDF document intelligent word segmentation method based on content structure perception, and belongs to the technical field of computer information.The PDF document intelligent word segmentation method comprises the steps that firstly, by fusing layout analysis and multi-tool analysis, multiple types of content such as title levels, ordinary tables, characters in pictures and tables in the pictures in a PDF are accurately recognized and extracted; secondly, providing a structure-perceived word segmentation strategy, and carrying out differentiation processing according to content types, namely, dividing semantic blocks by taking a title as a guide, taking a complete table as an independent semantic unit, and carrying out context association on a picture OCR (Optical Character Recognition) result; and finally outputting high-quality knowledge fragments rich in metadata such as levels and types. According to the method, the information integrity and semantic accuracy of the PDF document during construction of the large model knowledge base can be remarkably improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer information technology, and in particular to a PDF document intelligent word segmentation method based on content structure perception. BACKGROUND

[0002] With the popularity of digital documents, PDF has become the mainstream format for storing and transmitting knowledge. How to effectively transform the unstructured or semi-structured content in PDF into knowledge fragments that can be utilized by large models (such as RAG systems) has become a key challenge. Currently, the industry generally uses general word segmentation methods such as fixed-length sliding windows to simply segment PDF text. Such methods have significant defects:

[0003] 1. Serious structural damage: fixed segmentation will ruthlessly split the natural logical structure of the document, especially cutting a complete table into fragments and scattering them in multiple fragments, resulting in complete loss of core semantic information such as table row and column relationships and data comparisons.

[0004] 2. Semantic context is broken: simple word segmentation can easily break the strong semantic association between titles and main text, and between images / tables and their description text, resulting in incomplete semantic knowledge fragments.

[0005] 3. Weak mixed content processing capability: traditional methods are difficult to effectively process mixed text, tables, and images (especially tables in images) in PDF, and cannot achieve integrated information extraction and association.

[0006] Therefore, there is an urgent need in the art for a word segmentation method that can understand and maintain the internal logical structure of the document and intelligently process multiple types of content within it. SUMMARY

[0007] To solve the above technical problems, the present application provides a PDF document intelligent word segmentation method based on content structure perception to achieve the following purposes:

[0008] 1. Accurate content identification: accurately identify and extract key content such as titles, ordinary tables, text within images, and tables within images in PDF documents.

[0009] 2. Structure-preserving word segmentation: innovate a word segmentation strategy that follows the inherent logical structure of the document to ensure the integrity of the table and the semantic association between the title-main text-diagram.

[0010] 3. Output high-quality fragments: generate knowledge fragments rich in semantic information, complete structure, and rich metadata to provide high-quality knowledge for downstream large model applications.

[0011] The technical solution of the present application is:

[0012] A content-structure-aware intelligent word segmentation method for PDF documents, with the following specific steps:

[0013] 1) Structure analysis and content extraction: Analyze the layout of PDF documents, identify and extract the title hierarchy, table data and text content in image areas;

[0014] 2) Structure-aware word segmentation: Based on the extraction results of step 1), a differentiated word segmentation strategy is adopted, in which the identified title is used as the boundary to divide semantic blocks, and each identified complete table is treated as an independent and indivisible semantic unit;

[0015] 3) Metadata encapsulation: Add metadata to each knowledge fragment obtained after word segmentation. The metadata includes hierarchical path information that indicates the logical position of the fragment in the document, as well as content type information.

[0016] Furthermore,

[0017] Perform document layout analysis and region segmentation, including

[0018] Region detection and localization: Through a deep learning layout analysis model, the document page is parsed to detect and locate all possible content region bounding boxes;

[0019] Region type recognition: For each detected region, identify its functional type and classify it into title region, text paragraph region, ordinary table region, and image region;

[0020] Spatial relationship establishment: Based on the location and type of each region, analyze and establish the spatial location relationship and logical hierarchy relationship between them;

[0021] Structured output: The above analysis results are output as a structured document layout diagram, clearly marking the coordinates, type and interrelationships of each area.

[0022] Furthermore,

[0023] Step 1) involves extracting table data, including parsing the native tables in the document and / or parsing the tables identified from the image region.

[0024] Parsing tables identified from image regions involves: first, locating table regions in the image using an object detection model; then, analyzing the region using a table structure recognition model; and finally, outputting structured table data.

[0025] Furthermore,

[0026] Title recognition and hierarchy construction: Analyze layout features such as font, font size, and position, combine rules or lightweight models to identify titles at all levels, and construct a tree-like outline structure for the document.

[0027] Regular table extraction: Call a dedicated table parsing library to extract the structured information of a text-based table and convert it into HTML code that retains row and column attributes and spanning row and column attributes;

[0028] Text extraction from images: Using a high-precision OCR engine, text content in image areas is recognized;

[0029] Table extraction from images: Combining object detection and table structure recognition models, the tables in the images are restored into structured HTML table data.

[0030] Furthermore,

[0031] In step 1), the text content in the image area is associated and integrated with the semantic block to which it belongs, which is divided by the title.

[0032] Furthermore,

[0033] The word segmentation strategy includes

[0034] Title-guided semantic block segmentation: Using the identified title as the main boundary, the paragraphs, lists, and image descriptions to which it belongs are aggregated into a complete and semantically coherent semantic block;

[0035] Tables are processed as independent units: each extracted complete table is treated as a minimum, indivisible semantic unit, and treated as a separate knowledge fragment.

[0036] Image-text association and calibration: Associate the text content extracted from the image OCR with its semantic block or the nearest title to ensure contextual coherence.

[0037] Furthermore,

[0038] The metadata in step 3) also includes: source document traceability information and fragment unique identifiers, wherein,

[0039] Hierarchical path metadata: Records the precise location of knowledge fragments in the document outline, used to quickly locate the logical chapters in which the fragments are located during retrieval, and to provide important context for understanding their semantics;

[0040] Content type metadata: Clearly identifies the content essence of knowledge fragments, with values ​​including: text: representing a plain text paragraph; table: representing a complete structured table; image_ocr: representing text content extracted from an image; enabling downstream systems to intelligently filter and process different types of content according to requirements;

[0041] Document source information: Records the original source information of this fragment, including the name of the source document, storage path or unique identifier;

[0042] Fragment unique identifier: Assign a globally unique identifier to each knowledge fragment.

[0043] The beneficial effects of this invention are

[0044] Improved word segmentation quality: It solves the problem of general word segmentation not being suitable for PDF documents, and the generated segments conform to human semantic cognition, greatly preserving and highlighting the logical structure of the document.

[0045] Lossless preservation of key information: The table cellization strategy ensures the integrity of table information, providing a solid foundation for accurate data analysis of large models.

[0046] High retrieval accuracy and utilization: Rich metadata provides powerful context and filtering dimensions for vector retrieval, enabling more accurate semantic retrieval and thus improving the accuracy of answers generated by large models.

[0047] High versatility: The generated high-quality knowledge fragments rich in metadata not only optimize the performance of the RAG system, but are also applicable to various natural language processing tasks such as knowledge graph construction, intelligent question answering, and document structure retrieval, thus improving the versatility of PDF knowledge mining. Attached Figure Description

[0048] Figure 1 This is a schematic diagram of the workflow of the present invention. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0050] This invention provides a content structure-aware intelligent word segmentation method for PDF documents. First, by integrating layout analysis and multi-tool parsing, it identifies and extracts the content of heading levels, ordinary tables, text within images, and tables within images in the PDF. Second, it proposes a structure-aware word segmentation strategy, which performs differentiated processing based on content type. Specifically, it divides semantic blocks based on headings, treats complete tables as independent semantic units, and performs contextual association on image OCR results. Finally, it outputs high-quality knowledge fragments rich in hierarchy and type.

[0051] The specific technical solution is as follows:

[0052] (I) Document Layout Analysis and Region Segmentation

[0053] 1. Region Detection and Localization: Using a deep learning layout analysis model, the document page is parsed to detect and locate all possible content region bounding boxes.

[0054] 2. Region type recognition: For each detected region, identify its functional type and classify it into title region, text paragraph region, ordinary table region, image region, etc.

[0055] 3. Establishing spatial relationships: Based on the location and type of each region, analyze and establish the spatial positional relationships (such as vertical, containment, adjacency, etc.) and logical hierarchical relationships between them.

[0056] 4. Structured Output: The above analysis results are output as a structured document layout diagram, which clearly marks the coordinates, type and interrelationships of each area.

[0057] (II) Perception and Extraction of Content Structure

[0058] 1. Heading Recognition and Hierarchical Construction: Analyze layout features such as font, font size, and position, and combine rules or lightweight models to identify headings at all levels (H1, H2, ...) and construct the document's tree-like outline structure.

[0059] 2. Ordinary table extraction: Call a dedicated table parsing library to extract the structured information of the text-mode table and convert it into HTML code that retains row and column attributes.

[0060] 3. Text extraction from images: Using a high-precision OCR engine, text content in image areas is recognized.

[0061] 4. Table extraction from images: Combining object detection (for locating table regions) and table structure recognition models, the tables in the image are restored into structured HTML table data.

[0062] (III) Structure-Aware Intelligent Word Segmentation Strategy

[0063] 1. Title-guided semantic block segmentation: The identified title is used as the main boundary, and its associated paragraphs, lists, image descriptions, and other content are aggregated into a complete and semantically coherent semantic block.

[0064] 2. Table unitization and independent processing: Each extracted complete table is regarded as a minimum indivisible semantic unit and treated as a separate knowledge fragment.

[0065] 3. Image-text association and calibration: The text content extracted from the image OCR is associated with its semantic block or the nearest title to ensure contextual coherence.

[0066] (iv) Metadata information encapsulation

[0067] 1. Hierarchical path metadata: Records the precise location of knowledge fragments within the document outline. Specifically, it is represented by a list of title paths, such as ["Chapter 3 Core Technologies", "3.1 System Architecture", "3.1.1 Component Description"]. This metadata is used to quickly locate the logical chapter where the fragment is located during retrieval and provides important context for understanding its semantics.

[0068] 2. Content Type Metadata: Clearly identifies the content nature of knowledge fragments. Its values ​​include, but are not limited to: text: representing a plain text paragraph; table: representing a complete structured table; image_ocr: representing text content extracted from an image. This metadata enables downstream systems (such as RAG) to intelligently filter and process different types of content according to their needs. For example, it can perform refined operations such as "retrieving only tables" or "excluding image text."

[0069] 3. Document Source Metadata: Records the original source information of this fragment. It must include at least the name of the source document, its storage path, or a unique identifier. This metadata ensures the traceability of knowledge and facilitates copyright management, data updates, and error tracking.

[0070] 4. Unique Fragment Identifier: Each knowledge fragment is assigned a globally unique identifier. This identifier serves as the fragment's primary key, ensuring precise location and operation during storage, retrieval, and management, and providing the technical foundation for version control, associative linking, and lifecycle management of knowledge fragments.

[0071] The above description is merely a preferred embodiment of the present invention and is used only to illustrate the technical solution of the present invention, and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A content-structure-aware intelligent word segmentation method for PDF documents, characterized in that, The specific steps are as follows: 1) Structure analysis and content extraction: Analyze the layout of PDF documents, identify and extract the title hierarchy, table data and text content in image areas; 2) Structure-aware word segmentation: Based on the extraction results of step 1), a differentiated word segmentation strategy is adopted, in which the identified title is used as the boundary to divide semantic blocks, and each identified complete table is treated as an independent and indivisible semantic unit; 3) Metadata encapsulation: Add metadata to each knowledge fragment obtained after word segmentation. The metadata includes hierarchical path information that indicates the logical position of the fragment in the document, as well as content type information.

2. The method according to claim 1, characterized in that, Perform document layout analysis and region segmentation, including Region detection and localization: Through a deep learning layout analysis model, the document page is parsed to detect and locate all possible content region bounding boxes; Region type recognition: For each detected region, identify its functional type and classify it into title region, text paragraph region, ordinary table region, and image region; Spatial relationship establishment: Based on the location and type of each region, analyze and establish the spatial location relationship and logical hierarchy relationship between them; Structured output: The above analysis results are output as a structured document layout diagram, clearly marking the coordinates, type and interrelationships of each area.

3. The method according to claim 1, characterized in that, Step 1) involves extracting table data, including parsing the native tables in the document and / or parsing the tables identified from the image region.

4. The method according to claim 3, characterized in that, Parsing tables identified from image regions involves: first, locating table regions in the image using an object detection model; then, analyzing the region using a table structure recognition model; and finally, outputting structured table data.

5. The method according to claim 1, characterized in that, Title recognition and hierarchy construction: Analyze layout features such as font, font size, and position, combine rules or lightweight models to identify titles at all levels, and construct a tree-like outline structure for the document.

6. The method according to claim 4, characterized in that, Regular table extraction: Call a dedicated table parsing library to extract the structured information of a text-based table and convert it into HTML code that retains row and column attributes and spanning row and column attributes; Image text extraction: Using a high-precision OCR engine, the text content in the image area is recognized; Table extraction from images: Combining object detection and table structure recognition models, the tables in the images are restored into structured HTML table data.

7. The method according to claim 4, characterized in that, In step 1), the text content in the image area is associated and integrated with the semantic block to which it belongs, which is divided by the title.

8. The method according to claim 1, characterized in that, The word segmentation strategy includes Title-guided semantic block segmentation: Using the identified title as the main boundary, the paragraphs, lists, and image descriptions to which it belongs are aggregated into a complete and semantically coherent semantic block; Tables are processed as individual units: each extracted complete table is treated as a minimum, indivisible semantic unit, and treated as a separate knowledge fragment. Image-text association and calibration: Associate the text content extracted from the image OCR with its semantic block or the nearest title to ensure contextual coherence.

9. The method according to claim 1, characterized in that, The metadata in step 3) also includes: source document traceability information and fragment unique identifiers, wherein, Hierarchical path metadata: Records the precise location of knowledge fragments in the document outline, which is used to quickly locate the logical chapter where the fragment is located during retrieval and to provide important context for understanding its semantics; Content type metadata: Clearly identifies the content essence of knowledge fragments, with values ​​including: text: representing a plain text paragraph; table: representing a complete structured table; image_ocr: representing text content extracted from an image; enabling downstream systems to intelligently filter and process different types of content according to requirements; Document source information: Records the original source information of this fragment, including the name of the source document, storage path or unique identifier; Fragment unique identifier: Assign a globally unique identifier to each knowledge fragment.

Images