Document structure extraction and model training method and device, equipment and medium

Abstract

The invention discloses a document structure extraction and model training method and device, equipment and a medium, and relates to the technical field of artificial intelligence and computer vision. The method comprises the following steps: constructing a special training data set containing data of at least two document understanding tasks (including optical character recognition, layout analysis, text positioning, regional text extraction, image description and chart title generation), and a fine tuning data set for converting a document image into a machine-readable structured text format; constructing a multi-modal large model comprising a shape adaptive cutting module, a visual encoder, a visual token compression module, a modal connector and a language decoder; pre-training the model by using the special training data set to jointly learn various document understanding tasks; and performing fine tuning on the pre-trained model by using the fine tuning data set, and adapting to a document structure extraction task to obtain a document structure extraction model. By means of the technical scheme, efficient and accurate document structure extraction can be achieved.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and computer vision technology, and in particular to a method, apparatus, device and medium for document structure extraction and model training. Background Technology

[0002] Document Structured Extraction (DSE) aims to convert unstructured documents (such as PDFs) into machine-readable structured text formats (such as Markdown). This technology plays a crucial role in knowledge base construction and enhanced retrieval generation, and has become an important research direction for intelligent document understanding. Real-world PDFs (such as academic papers and office documents) are rich in content and complex in structure, often containing various information formats such as text, charts, tables, formulas, headers, and footers, and frequently exhibiting inconsistent page layouts and styles, significantly increasing the difficulty of structured extraction. Although existing methods have made some progress in document structure extraction, they still face multiple challenges in practical applications: First, models rely on high-quality, cross-domain labeled data during pre-training, but existing public datasets are still insufficient in terms of scale, format coverage, and language diversity; second, pre-training task design is relatively simple, lacking sufficient modeling of layout relationships, cross-modal alignment, and semantic hierarchy in structured documents; furthermore, complex document structures (such as multi-column layouts and mixed text and images) and style differences in different language environments also place higher demands on the model's generalization ability. Therefore, when faced with real PDF documents with diverse formats and complex semantics, existing methods often struggle to achieve accurate parsing of structure and semantics, especially in terms of cross-page information integration and content hierarchy restoration, where performance remains unstable. Summary of the Invention

[0003] In view of this, the purpose of this invention is to provide a document structure extraction and model training method, apparatus, device, and medium that can quickly and accurately convert complex PDFs into machine-readable Markdown, solving the problems of low efficiency, poor adaptability, and inaccurate parsing of traditional methods. The specific solution is as follows:

[0004] Firstly, this application discloses a training method for a document structure extraction model, comprising:

[0005] Construct a specialized training dataset and a fine-tuning dataset; the specialized training dataset includes data for various document understanding tasks, including at least two of optical character recognition, layout analysis, text localization, region text extraction, image description, and chart title generation; the fine-tuning dataset includes data for a document structure extraction task, which converts document images into machine-readable structured text format;

[0006] Construct a multimodal large model; the multimodal large model includes a shape adaptive pruning module, a visual encoder, a visual token compression module, a modal connector, and a language decoder;

[0007] The multimodal large model is pre-trained using the specialized training dataset to jointly learn multiple document understanding tasks.

[0008] The pre-trained multimodal large model is fine-tuned using the fine-tuning dataset to adapt the multimodal large model to the document structure extraction task, thereby obtaining the document structure extraction model.

[0009] Optionally, construct a specialized training dataset, including:

[0010] For the document understanding task of optical character recognition, the first document image is pre-annotated using an optical character recognition tool, and the pre-annotation results are manually corrected; a text detection model is trained using the corrected labels, and the trained text detection model is used to process the second document image to generate corresponding target text detection boxes; the text content in the target text detection boxes is extracted using a document processing library to construct an optical character recognition task dataset.

[0011] For the document understanding task of layout analysis, multiple open-source layout datasets are integrated, and samples containing semantic elements are selected from the open-source layout datasets to construct a layout analysis task dataset; the semantic elements include at least one of headings, body paragraphs, table cells, and image regions;

[0012] For the document understanding tasks of text localization and region text extraction, text blocks containing coordinate boxes are extracted from a third document image using the document processing library to construct a text localization task dataset with text block-coordinate box mapping and a region text extraction task dataset with coordinate box-text block mapping.

[0013] For the document understanding task of image description, multiple image description datasets are obtained. Natural images in the image description datasets are scaled and randomly inserted into a fourth document image. The original text descriptions of the image description datasets are used as supervision labels to construct an image description task dataset.

[0014] For the document comprehension task of generating chart titles, a preset toolkit is used to extract the coordinate boxes of the chart regions and the corresponding titles in the fifth document image in batches, so as to construct a chart title generation task dataset that associates charts with titles.

[0015] Optionally, construct a fine-tuning dataset, including:

[0016] Batch acquisition of unstructured documents on the target platform;

[0017] The document parsing tool automatically generates a sixth document image and a first data pair of Markdown format tags corresponding to the page from an unstructured English document.

[0018] The source code of the English unstructured document was machine translated into Chinese and then recompiled to generate a Chinese unstructured document.

[0019] The Chinese unstructured document is automatically generated into a seventh document image and a second data pair of Markdown format tags corresponding to the page using the document parsing tool.

[0020] A fine-tuning dataset is constructed based on the first data pair and the second data pair.

[0021] Secondly, this application discloses a document structure extraction method, including:

[0022] The unstructured document to be processed is converted into a target document image, and the target document image is input into a document structure extraction model; the document structure extraction model is a model trained using the aforementioned document structure extraction model training method; the document structure extraction model includes a shape adaptive cropping module, a visual encoder, a visual token compression module, a modal connector, and a language decoder;

[0023] The target document image is dynamically segmented using the shape adaptive cropping module to generate a global graph and a sequence of subgraphs;

[0024] The global graph and the subgraph sequence are input into the visual encoder to extract the corresponding global visual features and local visual features;

[0025] The global visual features and local visual features are input into the visual token compression module so that the local visual features are fused into the global visual features through a cross-attention mechanism to obtain compressed visual features.

[0026] The compressed visual features and text instruction features are aligned and spliced ​​together using the modal connector to form a multimodal fusion feature;

[0027] The multimodal fusion features are input into the language decoder to generate a structured text format corresponding to the target document image.

[0028] Optionally, the step of dynamically segmenting the target document image using the shape adaptive cropping module to generate a global graph and a sequence of subgraphs includes:

[0029] A predefined grid set is invoked to determine the matching score of each grid with the target document image; the predefined grid set contains grid parameters with different aspect ratios;

[0030] Select the target grid parameter with the highest matching score, and based on the target grid parameter, generate a global map and at least one sub-map of the target size from the target document image.

[0031] Optionally, the step of inputting the global visual features and local visual features into the visual token compression module to fuse the local visual features into the global visual features through a cross-attention mechanism to obtain compressed visual features includes:

[0032] By using a cross-attention mechanism, the global visual features are used as query vectors, and the local visual features are used as key vectors and value vectors for feature fusion, so as to fuse the local visual features into the global visual features and obtain the fused global features.

[0033] Perform a convolution operation with a preset kernel size on the fused global features to obtain compressed visual features.

[0034] Optionally, inputting the multimodal fusion features into the language decoder to generate a structured text format corresponding to the target document image includes:

[0035] The multimodal fusion features are input into a pre-trained language model with a parameter size not exceeding a preset order of magnitude to generate a structured text format corresponding to the target document image; wherein, the structured text format includes heading levels, table structure, and formula format, and automatically filters headers, footers, and image elements.

[0036] Thirdly, this application discloses a training device for a document structure extraction model, comprising:

[0037] The dataset construction module is used to construct specialized training datasets and fine-tuning datasets. The specialized training datasets include data for various document understanding tasks, including at least two of optical character recognition, layout analysis, text localization, region text extraction, image description, and chart title generation. The fine-tuning datasets include data for document structure extraction tasks, which are used to convert document images into machine-readable structured text formats.

[0038] The model building module is used to build a multimodal large model; the multimodal large model includes a shape adaptive pruning module, a visual encoder, a visual token compression module, a modal connector, and a language decoder;

[0039] The pre-training module is used to pre-train the multimodal large model using the specialized training dataset to jointly learn multiple document understanding tasks.

[0040] The fine-tuning module is used to fine-tune the pre-trained multimodal large model using the fine-tuning dataset, so that the multimodal large model can be adapted to the document structure extraction task, thereby obtaining a document structure extraction model.

[0041] Fourthly, this application discloses an electronic device comprising a processor and a memory; wherein the memory is used to store a computer program, which is loaded and executed by the processor to implement the method described above.

[0042] Fifthly, this application discloses a computer-readable storage medium for storing a computer program; wherein the computer program, when executed by a processor, implements the method described above.

[0043] The beneficial technical effects of this application are as follows: By constructing specialized training datasets covering various document understanding tasks and fine-tuning datasets for document structure extraction, the model is ensured to acquire comprehensive document semantic and layout understanding capabilities during the pre-training stage, while focusing on structured output generation during the fine-tuning stage, significantly improving the model's generalization and accuracy in complex document scenarios. In the model architecture, the shape adaptive cropping module can adapt to inputs of arbitrary resolution through dynamic segmentation, preserving global layout and local details; the visual token compression module can efficiently fuse features through a cross-attention mechanism, significantly compressing computational resources while maintaining information integrity, thus improving training and inference efficiency. The organic combination of pre-training and fine-tuning enables the model to possess both general document understanding capabilities and task-specific adaptability. The final generated document structure extraction model can efficiently and accurately convert document images into machine-readable structured formats, solving problems such as insufficient data diversity, low model efficiency, and inaccurate structure parsing in existing technologies, providing reliable technical support for intelligent document processing.

[0044] Furthermore, the document structure extraction model training device, document structure extraction method, equipment, and medium provided in this application correspond to the above-mentioned document structure extraction model training method and have the same effect. Attached Figure Description

[0045] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0046] Figure 1 This is a flowchart illustrating the training method for a document structure extraction model disclosed in this application.

[0047] Figure 2 This is a schematic diagram of a layout analysis pre-training task disclosed in this application;

[0048] Figure 3 This is a schematic diagram of an optical character recognition pre-training task disclosed in this application;

[0049] Figure 4 This is a schematic diagram of a pre-training task for text localization and region text extraction disclosed in this application;

[0050] Figure 5 This is a schematic diagram of an image description pre-training task disclosed in this application;

[0051] Figure 6 This is a schematic diagram of a pre-training task for generating chart titles as disclosed in this application;

[0052] Figure 7 This is a schematic diagram of a document structure extraction and fine-tuning task disclosed in this application;

[0053] Figure 8 This is a flowchart of a document structure extraction method disclosed in this application;

[0054] Figure 9 This is a schematic diagram of a shape adaptive clipping module disclosed in this application;

[0055] Figure 10 This is a diagram of a document structure extraction model architecture disclosed in this application;

[0056] Figure 11 This is a schematic diagram of the training device structure for a document structure extraction model disclosed in this application;

[0057] Figure 12 This is a structural diagram of an electronic device disclosed in this application. Detailed Implementation

[0058] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0059] PDF documents in daily work are typical "unstructured" data, and machines cannot directly understand their internal logic. Therefore, it is necessary to convert unstructured documents into machine-readable structured text formats. In recent years, methods based on Multi-modal Large Models (MLLM) have become a research hotspot, achieving end-to-end modeling of document structure by fusing image and text information. Representative methods include: the LayoutLM series, which introduces text position encoding to jointly model text and layout information, achieving excellent performance in tasks such as document classification and entity recognition, but it relies on the accuracy of the pre-processor OCR module and has limited ability to integrate cross-page information; Donut adopts an end-to-end Transformer architecture without OCR, supports multi-task learning, and has strong generalization ability, but its performance in structural hierarchy restoration is unstable; Pix2Struct is based on a generative architecture of image-to-structure sequence, effectively capturing image-text relationships, but there is still room for optimization in terms of structural accuracy and generation efficiency; TILT adopts a joint image-text Transformer model, which has advantages in multimodal alignment, but has a large training overhead; Graph Neural Network (GNN) methods show potential in complex layout parsing by modeling the structural relationships between document elements, but have high computational costs and are difficult to scale to large-scale documents.

[0060] While current large-scale model-based methods are powerful, they suffer from three major drawbacks: 1. The pre-training task design is too simplistic, focusing on training the model to perform only one task (e.g., recognizing only text), failing to adequately model key features in structured documents. This includes internal layout relationships, alignment logic of cross-modal information (e.g., text and images), and semantic hierarchy of document content, limiting the model's ability to deeply understand document structure. 2. Existing multimodal large-scale model-based document structure extraction methods have a large number of parameters and an excessive number of visual tokens (equivalent to small fragments of an image) per image, resulting in low model efficiency and high memory consumption. 3. The quantity and diversity of training data are insufficient. Existing public datasets have significant limitations in scale, format coverage, and language diversity, leading to a lack of sufficient and comprehensive data support during the model pre-training stage, making it difficult to adapt to document structure extraction needs in different scenarios.

[0061] To this end, this application provides a training scheme for a document structure extraction model, which can systematically enhance the model's core capabilities such as document semantic understanding and layout awareness, and demonstrate better learning efficiency and performance in fine-tuning tasks; it can greatly improve the model's running efficiency and reduce memory usage; and it can improve the model's generalization ability in complex scenarios.

[0062] This invention discloses a training method for a document structure extraction model, see [link to relevant documentation]. Figure 1 As shown, the method includes:

[0063] Step S11: Construct a specialized training dataset and a fine-tuning dataset; the specialized training dataset includes data for various document understanding tasks, including at least two of optical character recognition, layout analysis, text localization, region text extraction, image description, and chart title generation; the fine-tuning dataset includes data for document structure extraction tasks, which are used to convert document images into machine-readable structured text formats.

[0064] To address the issue of insufficient quantity and diversity of training data, current publicly available datasets suffer from significant shortcomings in scale, format coverage, and language diversity. This results in a lack of sufficient and comprehensive data support during the model pre-training stage, making it difficult to adapt to document structure extraction requirements in different scenarios. In this application, a unique data construction method generates a high-quality, cross-language pre-training and document structure extraction task training dataset with tens of millions of records, providing comprehensive data support for model training and improving the model's generalization ability in complex scenarios.

[0065] In this embodiment, a large-scale collection of Chinese and English PDF documents was obtained from the arXiv website and other open-source platforms. Specialized training datasets were then constructed to meet the specific needs of different document understanding tasks during pre-training. These specialized training datasets include six types of task-specific data: Optical Character Recognition (OCR) task data (including document images and corresponding text box coordinates and text content), layout analysis task data (including document images and coordinates and type labels for text, titles, charts, etc.), text localization task data (including document images and text-coordinate box mappings), region text extraction task data (including document images and coordinate box-text mappings), image description task data (including natural images in the document and corresponding text descriptions), and chart title generation task data (including chart images in the document and corresponding chart and table titles).

[0066] Meanwhile, addressing the scarcity of Chinese datasets, an innovative data augmentation strategy was adopted to construct a fine-tuned dataset for converting document images into machine-readable structured text formats (such as Markdown). Specifically, English arXiv papers were used to generate image-Markdown tag pairs using automated tools. English LaTeX source code was translated into Chinese, recompiled to generate Chinese PDFs, and then Chinese image-Markdown tag pairs were generated, forming a fine-tuned dataset that supports cross-language processing. The Markdown tags can contain structured information such as heading levels, table structures, and formula formats.

[0067] As can be seen, by constructing a large-scale, wide-ranging dataset of Chinese and English PDF documents in this application embodiment, which contains the data required for both document understanding tasks during model pre-training and document structure extraction tasks during model fine-tuning, this dataset can cover multiple industries and application scenarios, significantly improving the diversity of training data and the generalization ability of the model.

[0068] Step S12: Construct a multimodal large model; the multimodal large model includes a shape adaptive pruning module, a visual encoder, a visual token compression module, a modal connector, and a language decoder.

[0069] Secondly, the model architecture is constructed. The constructed multimodal large model includes a shape adaptive pruning module, a visual encoder, a visual token compression module, a modal connector, and a language decoder.

[0070] The system comprises the following modules: a shape-adaptive cropping module for dynamically segmenting the input document image, generating a global map and sub-maps of uniform size to preserve global layout and local details; a visual encoder for extracting features from the global and sub-maps; a visual token compression module for fusing sub-map features into global map features using a cross-attention mechanism to compress the number of visual tokens; a modal connector for dimensionally aligning the compressed visual features with text features; and a language decoder for generating text sequences based on the aligned features. The specific configurations of each module will be detailed in later embodiments and will not be elaborated upon here.

[0071] Step S13: Pre-train the multimodal large model using the specialized training dataset to jointly learn multiple document understanding tasks.

[0072] The current pre-training task design is too simplistic and fails to adequately model the key features of structured documents, including internal layout relationships, alignment logic of cross-modal information (such as text and images), and semantic hierarchy of document content. This limits the model's ability to deeply understand document structure. In this application, six document understanding tasks are specifically designed for pre-training, systematically enhancing the model's core capabilities such as document semantic understanding and layout awareness. This results in superior learning efficiency and performance in fine-tuning tasks.

[0073] In this embodiment, a multimodal large-scale model is pre-trained based on a specialized training dataset. To comprehensively enhance the fundamental capabilities of the document understanding multimodal large-scale model, a series of multi-dimensional pre-training tasks are designed. A multi-task joint learning mechanism strengthens the model's deep understanding of document structure and semantics, building a solid technical foundation for downstream tasks. Pre-training includes multiple document understanding tasks executed in parallel, including at least two of optical character recognition, layout analysis, text localization, region text extraction, image description, and chart title generation. This strengthens the model's ability to model document structure and semantics from multiple levels, laying a solid foundation for subsequent PDF-to-Markdown fine-tuning tasks.

[0074] It should be noted that the model training strategy utilizes 16 NVIDIA A800 GPUs for computation. During the pre-training phase, to fully explore the model's fundamental capabilities and achieve global parameter optimization, a full-parameter training strategy is employed. This involves releasing the parameters of all model modules and performing end-to-end joint training on six pre-training tasks to enhance the model's ability to model document layout, cross-modal alignment, and semantic hierarchy. The parameters are set as follows: epoch=3, learning rate lr=1e-4, batch_size=1024.

[0075] Step S14: Use the fine-tuning dataset to fine-tune the pre-trained multimodal large model so that the multimodal large model is adapted to the document structure extraction task, thereby obtaining the document structure extraction model.

[0076] After pre-training, the model is fine-tuned using the fine-tuning dataset corresponding to the document structure extraction task, so that the model is adapted to the output task of converting document images into a structured format.

[0077] In this embodiment, during the model fine-tuning stage, the parameters of the visual encoder, shape adaptive cropping module, visual token compression module, and modal connector are fixed based on the pre-trained model parameters, and only the language decoder is trained. The fine-tuning dataset is used to adapt the PDF to Markdown document structure extraction task, and the training parameters are set as follows: epoch=3, learning rate lr=1e-5, batch_size=1024, to obtain the trained lightweight document structure extraction model.

[0078] The beneficial technical effects of this application are as follows: By constructing specialized training datasets covering various document understanding tasks and fine-tuning datasets for document structure extraction, the model is ensured to acquire comprehensive document semantic and layout understanding capabilities during the pre-training stage, while focusing on structured output generation during the fine-tuning stage, significantly improving the model's generalization and accuracy in complex document scenarios. In the model architecture, the shape adaptive cropping module can adapt to inputs of arbitrary resolution through dynamic segmentation, preserving global layout and local details; the visual token compression module can efficiently fuse features through a cross-attention mechanism, significantly compressing computational resources while maintaining information integrity, thus improving training and inference efficiency. The organic combination of pre-training and fine-tuning enables the model to possess both general document understanding capabilities and task-specific adaptability. The final generated document structure extraction model can efficiently and accurately convert document images into machine-readable structured formats, solving problems such as insufficient data diversity, low model efficiency, and inaccurate structure parsing in existing technologies, providing reliable technical support for intelligent document processing.

[0079] Based on the foregoing embodiments, this application describes the specific implementation details of constructing specialized datasets and corresponding document understanding tasks, as well as constructing fine-tuning datasets and corresponding document structure extraction tasks. Document understanding tasks include optical character recognition, layout analysis, text localization, region text extraction, image description, and chart title generation, etc.

[0080] (1) For the document understanding task of layout analysis, multiple open-source layout datasets are integrated, and samples containing semantic elements are selected from the open-source layout datasets to construct a layout analysis task dataset; the semantic elements include at least one of title, body paragraph, table cell, and image region.

[0081] In the document understanding task of layout analysis, six mainstream open-source layout datasets—CDLA, Docgenome, Publaynet, Doclaynet, Docbank, and M6Doc—were integrated to comprehensively cover typical document structure elements in academic and office scenarios. Detailed statistical information for each dataset (such as sample size and label distribution) is shown in Table 1.

[0082] Table 1. Statistics of Pre-trained Dataset for Layout Analysis

[0083]

[0084] Accordingly, to enhance the pre-trained model's ability to deeply analyze document layout structure, a pre-training task based on layout analysis was designed. The core objective is to achieve accurate spatial coordinate localization and hierarchical relationship modeling of various semantic elements (such as headings, body paragraphs, table cells, and image regions) in document images. Related text instructions, model output examples, and layout analysis visualization results are shown below. Figure 2 As shown.

[0085] (2) For the document understanding task of optical character recognition, the first document image is pre-annotated using an optical character recognition tool and the pre-annotation results are manually corrected; the text detection model is trained using the corrected labels and the trained text detection model is used to process the second document image to generate the corresponding target text detection box; the text content in the target text detection box is extracted through the document processing library to construct the optical character recognition task dataset.

[0086] In document understanding tasks using Optical Character Recognition (OCR), to obtain higher-quality pre-training data, a large number of first document images are pre-annotated using OCR tools such as PaddleOCR to obtain preliminary text detection boxes (at this stage, the detection boxes may have issues such as missed detections, positioning errors, and false detections, such as missing small text, detection boxes not completely enclosing text, or labeling non-text areas as text). Then, the annotation results are optimized through manual correction (supplementing missed text boxes, adjusting text boxes with positioning errors, and deleting false detection text boxes). The manually corrected labels are then input into the DB text detection model for training, allowing the model to learn the ability to "identify text regions from document image pixels and output accurate detection boxes." Finally, a text detection model with sufficient accuracy is obtained, which can directly output high-quality detection boxes for any document image (with positioning errors far lower than those of manual pre-annotation). At this point, the trained text detection model is used to process the second document image to generate the corresponding target text detection boxes. It is understandable that the second document image and the first document image can be the same set of images or different sets of images. In one feasible implementation, in order to ensure that the text detection model can have stable detection accuracy and the generated OCR dataset is more generalizable, the second document image can be a new document image that is different from or contains the first document image. Finally, the text in the detection box is extracted by the document processing library, thereby constructing a high-quality OCR task dataset.

[0087] For example, the document processing library is the Fitz library. It doesn't inherently possess the ability to "identify text location"; it must first know "which region of text to extract" (i.e., the coordinates of the detection boxes) to accurately extract the corresponding content. The detection boxes output by the trained text detection model provide the Fitz library with the "extraction range." Applying the trained text detection model to a new PDF document image, the model outputs detection boxes (including coordinates) for all text in the image. Calling the Fitz library's text extraction interface, with the coordinates of each detection box as input parameters, the Fitz library locates the corresponding rectangular region in the image based on these coordinates and extracts the text content within that region. This ultimately forms a triplet of data: "document image → detection box (coordinates) → text content," which is the high-quality training dataset required for the OCR task.

[0088] Accordingly, for OCR document understanding tasks, during pre-training, it is used to accurately convert visual pixel information in document images into text sequences and their corresponding coordinates, building a basic mapping capability between vision and text, such as... Figure 3 As shown. The input instruction for the OCR task is "OCR the image," and the output adopts a structured format: with text lines as the basic unit, each line of the result contains the bounding box information formed by the coordinates of the top-left pixel [x0, y0] and the bottom-right pixel [x1, y1], as well as the corresponding text content within the bounding box. As a core pre-training task for multimodal document understanding, the OCR task achieves accurate conversion from unstructured images to structured text information, laying a solid foundation for subsequent tasks such as semantic parsing, key information extraction, and structural analysis.

[0089] (3) For document understanding tasks involving text localization and regional text extraction, text blocks containing coordinate boxes are extracted from the third document image through the document processing library to construct a text localization task dataset with text block-coordinate box mapping and a regional text extraction task dataset with coordinate box-text block mapping.

[0090] For document understanding tasks involving text localization (Text2Box) and region text extraction (Box2Text), the datasets are generated automatically: using the `get_text_blocks()` function from the Fitz library, text blocks and their corresponding bounding boxes are directly extracted from PDF documents. This constructs a text localization dataset with a "text content-bounding box" mapping and a region text extraction dataset with a "bounding box-text content" mapping. This bidirectional "coordinate-text" mapping dataset provides data support for the model to master text region localization and content extraction capabilities.

[0091] It is understandable that the PDF document being processed at this time is a third document image, which is a set of input images used for text localization and region text extraction tasks. It can come from the same batch or different PDF document resources as the aforementioned document images. Its core function is to extract text blocks and coordinate boxes through the document processing library to construct a bidirectional mapping dataset.

[0092] Correspondingly, during model pre-training, the text localization task primarily focuses on accurately identifying the spatial coordinates of given text from document images, such as... Figure 4 As shown, the core objective of this task design is "text content → spatial coordinates," meaning that for a specific text fragment in a document image, the model needs to predict its precise bounding box coordinates within the image. This design improves the model's understanding of the spatial distribution patterns of text, provides reliable spatial localization support for cross-modal semantic alignment, and promotes the close integration of text information and image content.

[0093] Furthermore, as a powerful complement to text localization tasks, region text extraction tasks focus on accurately extracting and recognizing text content from a given image region, thereby constructing an inverse mapping relationship of "spatial region → text content," such as... Figure 4 As shown. Unlike "text content localization," which focuses on prediction from semantics to space, this task emphasizes reconstructing textual information from visual space, thus improving the model's ability to understand and analyze the semantics of local regions.

[0094] (4) For the document understanding task of image description, multiple image description datasets are obtained, the natural images in the image description datasets are scaled and randomly inserted into the fourth document image, and the original text descriptions of the image description datasets are used as supervision labels to construct the image description task dataset.

[0095] The image captioning document understanding task dataset comes from four specialized image captioning datasets on the Hugging Face platform: LLaVA-ReCap-CC3M, LLaVA-ReCap-118K, LLaVA-ReCap-558K, and laion-coco-nllb. Original images are scaled and randomly inserted into various fourth-document images. The model is trained using their inherent descriptive text as supervised labels, enabling it to generate accurate text descriptions for natural images of specified regions. Understandably, these fourth-document images can overlap with or be independent of other task images; supervised labels are generated by inserting natural images into these document images. This large-scale, high-quality specialized dataset provides ample data support for the model to learn the correlation between natural image content understanding and text generation.

[0096] Correspondingly, the image description task aims to generate relevant descriptions of natural images of specific regions in a document image, thereby enhancing the model's cross-modal alignment and visual detail recognition capabilities, and improving its spatial structure understanding capabilities, such as... Figure 5 As shown, this design, through training on the strong correlation between "natural images and text," significantly improves the model's ability to semantically parse complex visual elements in documents, providing crucial support for cross-modal content understanding.

[0097] (5) For the document understanding task of generating the chart title, use a preset toolkit to extract the coordinate frame of the chart area and the corresponding title in the fifth document image in batches, so as to construct a chart title generation task dataset that associates charts and titles.

[0098] The fifth document image is a set of input images used for the chart title generation task. It can come from the same batch or different PDF document resources (such as downloaded arXiv English papers). Its core function is to serve as a data source, extracting chart coordinates and title information through the toolkit to achieve phased differentiation of the task process. In actual implementation, this image set can overlap with other task images or be independently allocated to improve data construction efficiency.

[0099] Specifically, the dataset for the document understanding task of figure / table captioning is constructed based on downloaded English papers from arXiv. Pre-set toolkits (such as pdffigures2 Java) are used to generate the coordinate frame information of images and tables in batches, while extracting the corresponding figure and table names. Finally, a pre-trained dataset for figure / table captioning is constructed to achieve accurate association modeling between figures and captions.

[0100] Correspondingly, for document images containing both image and table elements, the chart title generation task focuses on generating precise corresponding chart and table titles for images and tables at specified locations. This strengthens the model's ability to learn the logical relationship between chart content and document context, such as... Figure 6 As shown.

[0101] Through the construction of the aforementioned specialized datasets and the collaborative optimization of pre-training tasks, the model's capabilities in multiple dimensions, such as visual perception, text understanding, and structural parsing, have been comprehensively improved, providing stronger foundational support for subsequent downstream applications such as document structure extraction and information retrieval.

[0102] Furthermore, based on the pre-trained multimodal large model, specific fine-tuning and optimization work is carried out for the downstream task of document structure extraction, such as... Figure 7As shown. This task requires accurately outputting all text in a document image, including the hierarchical relationship of headings, following the rule that "the more # symbols, the lower the level, with one # symbol representing a first-level heading, and so on." It also requires accurately presenting table structures and formula formats, while elements such as headers, footers, and images do not need to be included in the processing, ultimately generating machine-readable structured output content. Through fine-tuning, the model achieved precise adaptation from general document understanding to specific tasks, significantly improving the quality of structured PDF to Markdown generation.

[0103] For the document structure extraction task in the model fine-tuning process, the process of constructing the fine-tuning dataset includes:

[0104] Step 1: Batch acquire unstructured documents on the target platform;

[0105] Step 2: Use a document parsing tool to automatically generate a sixth document image and the first data pair of Markdown format tags corresponding to the page from the unstructured English document;

[0106] Step 3: Machine translate the source code of the English unstructured document into Chinese, and recompile to generate a Chinese unstructured document;

[0107] Step 4: The Chinese unstructured document is automatically generated into a seventh document image and a second data pair of Markdown format tags corresponding to the page using the document parsing tool;

[0108] Step 5: Construct a fine-tuning dataset based on the first data pair and the second data pair.

[0109] In the PDF to Markdown fine-tuning task, in one specific implementation, for the generation of English fine-tuning data, English PDF papers are obtained in batches from the arXiv website. A document parsing tool (such as the nougat project) is used to automatically convert each paper page by page to generate a single-page document image (i.e., the sixth document image). At the same time, corresponding page Markdown tags (including heading levels, table structure, and formula format) are generated to form "English document image-Markdown" data pairs, providing basic data support for model fine-tuning.

[0110] In another specific implementation, to address the scarcity of Chinese datasets and the generation of fine-tuned Chinese data, an innovative data augmentation strategy is adopted: the LaTeX source code of the arXiv English paper is translated into Chinese, preserving the formatting logic of formulas and tables; the translated LaTeX source code is recompiled to generate a Chinese PDF; and the nougat project workflow is followed to generate data pairs of Chinese document images (i.e., the seventh document image) and corresponding Markdown tags.

[0111] This method constructs a large-scale Chinese fine-tuning dataset, improving the linguistic diversity and coverage of the training data, and laying the foundation for enhancing the model's cross-linguistic generalization ability. Table 2 shows the dataset statistics for the other pre-training document understanding tasks and the statistics for the fine-tuning dataset.

[0112] Table 2 Statistics of Pre-training and Fine-tuning Datasets

[0113]

[0114] In this embodiment, during the construction of the document structure extraction model, firstly, training datasets for six pre-training tasks—document structure extraction, optical character recognition, layout analysis, image description, text localization, region text extraction, and chart title extraction—are constructed. Secondly, the model is pre-trained based on the designed lightweight document structure extraction model architecture and the constructed pre-training task training datasets. Then, the downstream tasks of document structure extraction are fine-tuned based on the pre-trained model and the constructed document structure extraction training datasets. Finally, the finely-tuned document structure extraction model is used for document structure extraction, which can accurately and quickly convert PDF document images into Markdown format. It is evident that, relying on tens of millions of high-quality labeled samples, a customized lightweight multimodal model architecture, and proprietary data processing and optimization methods, this invention ultimately trains a high-performance document structure extraction model. This model exhibits significant advantages in PDF to Markdown conversion tasks, particularly demonstrating good robustness and accuracy in complex structure recognition and cross-page information restoration, providing strong technical support for intelligent document processing.

[0115] This application discloses a document structure extraction method, see [link to relevant documentation]. Figure 8 As shown, the method includes:

[0116] Step S21: Convert the unstructured document to be processed into a target document image, and input the target document image into the document structure extraction model; the document structure extraction model is a model trained using the aforementioned document structure extraction model training method; the document structure extraction model includes a shape adaptive cropping module, a visual encoder, a visual token compression module, a modal connector, and a language decoder.

[0117] When using a document structure extraction model to extract structure from PDF documents, the process begins by acquiring the unstructured document to be processed and converting it page by page into original document images, which can support arbitrary resolutions. Then, the processed target document images are input into the document structure extraction model trained based on the aforementioned steps, and the model is used to process the target document images.

[0118] Step S22: Use the shape adaptive cropping module to dynamically segment the target document image to generate a global graph and a sequence of subgraphs.

[0119] In the model architecture, the shape adaptive cropping module is configured to receive document images of arbitrary resolution, as illustrated in the diagram below. Figure 9 As shown, by matching the optimal aspect ratio using a predefined grid and adapting to inputs of arbitrary resolution through dynamic segmentation, the system preserves both global layout and local details while avoiding information compression loss, laying the foundation for accurate feature extraction and semantic understanding.

[0120] Specifically, the process includes the following steps: calling a predefined grid set to determine the matching score of each grid with the target document image; the predefined grid set contains grid parameters with different aspect ratios; selecting the target grid parameter with the highest matching score, and generating a global map and at least one sub-map of the target size from the target document image based on the target grid parameter.

[0121] For example, the shape adaptive clipping module uses a predefined grid strategy to generate a uniform sequence of subgraphs and a global graph through the following steps:

[0122] 1. Grid matching and image segmentation: Match the optimal aspect ratio using a predefined grid set, calculate the score based on relevant indicators, and select the grid configuration g=(nh, nw) that maximizes the score, where nh and nw represent the number of sub-image segments in the height and width dimensions, respectively.

[0123] 2. Sub-image generation: Based on the selected grid parameters, scale the input image to (nh×H, nw×W) and divide it into nh×nw fixed-size sub-images (H is the height, W is the width);

[0124] 3. Global Context Construction: The original image is scaled to (H, W) to generate a global image, forming a complementary architecture of global graph and subgraph.

[0125] In one feasible implementation, to adapt to the typical aspect ratio distribution of document images in real-world scenarios, the module parameters are configured as follows: standard sub-image size: height H=1024, width W=768; maximum number of cropped blocks constraint: N=8 (i.e., nh×nw≤8). That is, the image is divided into no more than 8 sub-images of fixed size (height 1024, width 768) and 1 global image of the same size.

[0126] This parameter setting can match the aspect ratio of most PDF documents, avoid deformation compression caused by forced scaling during preprocessing, fully preserve detailed features, and is compatible with any input resolution. It preserves the overall layout and local details of the image through a complementary strategy of global and sub-images.

[0127] Step S23: Input the global graph and the subgraph sequence into the visual encoder to extract the corresponding global visual features and local visual features.

[0128] In this embodiment, a lightweight model architecture is designed, using SwinV2_B as the visual encoder. After multi-scale visual features are extracted and processed by the SwinV2 encoder, the number of image tokens is (H / 32, W / 32) × (N+1), where N is the number of sub-images segmented and 1 represents the global image.

[0129] Step S24: Input the global visual features and local visual features into the visual token compression module, so as to fuse the local visual features into the global visual features through the cross-attention mechanism to obtain the compressed visual features.

[0130] To further optimize computational efficiency, a visual token compression module is designed in this embodiment. This module significantly reduces the number of visual features while retaining global and local visual information, effectively improving the model's processing speed and memory usage efficiency.

[0131] In this embodiment, the visual token compression module fuses the local features of the subgraph into the global graph feature map through a cross-attention mechanism. Then, the global features enhanced by local features are input into the language decoder to achieve high-ratio visual token compression. The specific process is as follows: Figure 10 As shown. Specifically, the visual token compression module performs the following steps: using a cross-attention mechanism, the global visual features are used as query vectors, and the local visual features are used as key vectors and value vectors for feature fusion, so as to fuse the local visual features into the global visual features to obtain fused global features; and a convolution operation with a preset kernel size is performed on the fused global features to obtain compressed visual features.

[0132] In this embodiment, after the PDF document image passes through a visual encoder, the global graph features primarily encode the overall text layout information, while the subgraph features are responsible for capturing detailed text information. Based on this, the global graph features are used as the query, and the subgraph features are used as the key and value, respectively. A cross-attention mechanism is then used to achieve deep integration of textual semantic information.

[0133] After processing by the visual token compression module, the number of image tokens is compressed from (H / 32, W / 32)×(N+1) to (H / 32, W / 32). During this process, all sub-image features are fully integrated into the global image to form an enhanced global feature map, thereby achieving efficient visual token compression.

[0134] In one feasible implementation, to further enhance the horizontal feature aggregation capability, the module further compresses the number of visual tokens to (H / 32, W / 64) through a convolution operation with a kernel size of 1×2. With a width W=768 and a height H=1024, regardless of the original resolution of the input image, the number of visual tokens is uniformly compressed to (1024 / 32)×(768 / 64)=384.

[0135] As can be seen, the visual token compression module efficiently fuses features through a cross-attention mechanism, significantly compressing computational resources while maintaining information integrity, thereby improving training and inference efficiency.

[0136] Step S25: Align and splice the compressed visual features and text instruction features through the modal connector to form a multimodal fusion feature.

[0137] In this embodiment, a concise and efficient fully connected layer (MLP) is used as a modal connector to map visual features and text features to a unified semantic space, achieving accurate alignment of cross-modal features. After completing feature dimension matching and semantic calibration, the aligned visual features and text instruction features (including instructions converted to Markdown format) are sequentially concatenated to form multimodal fusion features, which are then input into a large language model for decoding output.

[0138] It should be noted that text instruction features refer to the fixed-dimensional numerical vectors converted from natural language instructions that clearly state the model's task objective (such as "convert to Markdown format" or "OCR the image") by a text encoder (the Transformer encoder that accompanies the language decoder Qwen2-0.5B). Its core function is to "issue task instructions to the model"—multimodal models need to simultaneously receive both "visual features of the document image" ("what to look at") and "text instruction features" ("what to do") to accurately execute the corresponding structured extraction task and avoid the model confusing the task objective (e.g., mistakenly executing "PDF to Markdown" as "simple OCR recognition").

[0139] Step S26: Input the multimodal fusion features into the language decoder to generate a structured text format corresponding to the target document image.

[0140] To improve the training and running efficiency of the multimodal large-scale model for document understanding, Qwen2-0.5B is used as the language decoder to receive multimodal fusion features and generate structured text. This model maintains a lightweight architecture (parameter size of only 0.5B) while possessing excellent semantic understanding and text generation capabilities, providing stable and efficient support for complex document parsing tasks. Specifically, the language decoder performs the following steps: inputting the multimodal fusion features into a pre-trained language model with a parameter size not exceeding a preset order of magnitude to generate a Markdown-formatted structured text corresponding to the target document image; wherein the structured text format includes heading levels (distinguished by the number of # symbols), table structure, and formula format, and automatically filters headers, footers, and image elements.

[0141] Finally, the Markdown results corresponding to each page of the PDF document are integrated in page number order to form a complete structured extraction result of the PDF document.

[0142] As can be seen, this application presents a lightweight model architecture for document structure extraction: using Swinv2 as the visual encoder, a fully connected layer as the modal connector, and Qwen2-0.5B as the language decoder, significantly improving operational efficiency while ensuring model performance. This application also proposes an efficient visual token compression module based on a cross-attention mechanism, fusing local information into the global feature map, thereby compressing the number of visual tokens in a single image to 384, greatly improving the model's inference efficiency and memory utilization. Furthermore, this application proposes a shape-adaptive cropping module suitable for document structure extraction, setting the sub-image and global image to a uniform size: width = 768, height = 1024, to adapt to the typical aspect ratio distribution of document images in real-world scenarios. This module receives original document images of arbitrary resolution and dynamically adjusts the segmentation strategy to retain global structural features and local fine-grained features while avoiding information compression loss, laying the foundation for accurate feature extraction and semantic understanding.

[0143] Accordingly, this application also discloses a training device for a document structure extraction model, see [link to relevant documentation]. Figure 11 As shown, the device includes:

[0144] The dataset construction module 11 is used to construct a specialized training dataset and a fine-tuning dataset; the specialized training dataset includes data for various document understanding tasks, including at least two of optical character recognition, layout analysis, text localization, region text extraction, image description, and chart title generation; the fine-tuning dataset includes data for a document structure extraction task, which is used to convert document images into machine-readable structured text format.

[0145] The model building module 12 is used to build a multimodal large model; the multimodal large model includes a shape adaptive pruning module, a visual encoder, a visual token compression module, a modal connector, and a language decoder;

[0146] Pre-training module 13 is used to pre-train the multimodal large model using the specialized training dataset to jointly learn multiple document understanding tasks.

[0147] The fine-tuning module 14 is used to fine-tune the pre-trained multimodal large model using the fine-tuning dataset, so that the multimodal large model can be adapted to the document structure extraction task, thereby obtaining a document structure extraction model.

[0148] For more detailed information on the working process of each of the above modules, please refer to the relevant content disclosed in the foregoing embodiments, which will not be repeated here.

[0149] Therefore, the above-described scheme in this embodiment, by constructing specialized training datasets covering various document understanding tasks and fine-tuning datasets for document structure extraction, ensures that the model acquires comprehensive document semantic and layout understanding capabilities during the pre-training stage, and focuses on generating structured outputs during the fine-tuning stage, significantly improving the model's generalization and accuracy in complex document scenarios. In the model architecture, the shape adaptive cropping module can adapt to inputs of arbitrary resolution through dynamic segmentation, preserving both global layout and local details; the visual token compression module can efficiently fuse features through a cross-attention mechanism, significantly compressing computational resources while maintaining information integrity, thus improving training and inference efficiency. The organic combination of pre-training and fine-tuning enables the model to possess both general document understanding capabilities and task-specific adaptability. The resulting document structure extraction model can efficiently and accurately convert document images into machine-readable structured formats, solving problems such as insufficient data diversity, low model efficiency, and inaccurate structure parsing in existing technologies, providing reliable technical support for intelligent document processing.

[0150] Furthermore, embodiments of this application also disclose an electronic device, Figure 12 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content of the diagram should not be construed as limiting the scope of this application.

[0151] Figure 12This is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of this application. Specifically, the electronic device 20 may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the methods disclosed in any of the foregoing embodiments. Furthermore, the electronic device 20 in this embodiment may specifically be a computer.

[0152] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0153] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk, or optical disk, etc. The resources stored on it can include an operating system 221, computer programs 222, and data 223, etc. The data 223 can include various types of data. The storage method can be temporary storage or permanent storage.

[0154] The operating system 221 is used to manage and control the various hardware devices on the electronic device 20 and the computer program 222, which may be Windows Server, Netware, Unix, Linux, etc. In addition to including computer programs capable of performing the methods executed by the electronic device 20 as disclosed in any of the foregoing embodiments, the computer program 222 may further include computer programs capable of performing other specific tasks.

[0155] Furthermore, this application also discloses a computer-readable storage medium, which includes random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disks, magnetic disks, optical disks, or any other form of storage medium known in the art. The computer program, when executed by a processor, implements the aforementioned method. Specific steps of this method can be found in the corresponding content disclosed in the foregoing embodiments, and will not be repeated here.

[0156] Furthermore, embodiments of this application also provide a computer program product, including a computer program / instructions that, when executed by a processor, implement any of the methods described above.

[0157] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0158] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0159] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0160] The above provides a detailed description of the document structure extraction and model training method, apparatus, device, and medium provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only intended to help understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A training method for a document structure extraction model, characterized in that, include: Construct a specialized training dataset and a fine-tuning dataset; the specialized training dataset includes data for various document understanding tasks, including at least two of optical character recognition, layout analysis, text localization, region text extraction, image description, and chart title generation; the fine-tuning dataset includes data for a document structure extraction task, which converts document images into machine-readable structured text format; Construct a multimodal large model; the multimodal large model includes a shape adaptive pruning module, a visual encoder, a visual token compression module, a modal connector, and a language decoder; The multimodal large model is pre-trained using the specialized training dataset to jointly learn multiple document understanding tasks. The pre-trained multimodal large model is fine-tuned using the fine-tuning dataset to adapt the multimodal large model to the document structure extraction task, thereby obtaining the document structure extraction model.

2. The training method for the document structure extraction model according to claim 1, characterized in that, Construct a specialized training dataset, including: For the document understanding task of optical character recognition, the first document image is pre-annotated using an optical character recognition tool, and the pre-annotation results are manually corrected; a text detection model is trained using the corrected labels, and the trained text detection model is used to process the second document image to generate corresponding target text detection boxes; the text content in the target text detection boxes is extracted using a document processing library to construct an optical character recognition task dataset. For the document understanding task of layout analysis, multiple open-source layout datasets are integrated, and samples containing semantic elements are selected from the open-source layout datasets to construct a layout analysis task dataset; the semantic elements include at least one of headings, body paragraphs, table cells, and image regions; For the document understanding tasks of text localization and region text extraction, text blocks containing coordinate boxes are extracted from a third document image using the document processing library to construct a text localization task dataset with text block-coordinate box mapping and a region text extraction task dataset with coordinate box-text block mapping. For the document understanding task of image description, multiple image description datasets are obtained. Natural images in the image description datasets are scaled and randomly inserted into a fourth document image. The original text descriptions of the image description datasets are used as supervision labels to construct an image description task dataset. For the document comprehension task of generating chart titles, a preset toolkit is used to extract the coordinate boxes of the chart regions and the corresponding titles in the fifth document image in batches, so as to construct a chart title generation task dataset that associates charts with titles.

3. The training method for the document structure extraction model according to claim 1, characterized in that, Constructing a fine-tuning dataset includes: Batch acquisition of unstructured documents on the target platform; The document parsing tool automatically generates a sixth document image and a first data pair of Markdown format tags corresponding to the page from an unstructured English document. The source code of the English unstructured document was machine translated into Chinese and then recompiled to generate a Chinese unstructured document. The Chinese unstructured document is automatically generated into a seventh document image and a second data pair of Markdown format tags corresponding to the page using the document parsing tool. A fine-tuning dataset is constructed based on the first data pair and the second data pair.

4. A document structure extraction method, characterized in that, include: The unstructured document to be processed is converted into a target document image, and the target document image is input into a document structure extraction model; the document structure extraction model is a model trained using the training method of the document structure extraction model as described in any one of claims 1 to 3; the document structure extraction model includes a shape adaptive cropping module, a visual encoder, a visual token compression module, a modal connector, and a language decoder; The target document image is dynamically segmented using the shape adaptive cropping module to generate a global graph and a sequence of subgraphs; The global graph and the subgraph sequence are input into the visual encoder to extract the corresponding global visual features and local visual features; The global visual features and local visual features are input into the visual token compression module so that the local visual features are fused into the global visual features through a cross-attention mechanism to obtain compressed visual features. The compressed visual features and text instruction features are aligned and spliced ​​together using the modal connector to form a multimodal fusion feature; The multimodal fusion features are input into the language decoder to generate a structured text format corresponding to the target document image.

5. The document structure extraction method according to claim 4, characterized in that, The step of dynamically segmenting the target document image using the shape adaptive cropping module to generate a global graph and a sequence of subgraphs includes: A predefined grid set is invoked to determine the matching score of each grid with the target document image; the predefined grid set contains grid parameters with different aspect ratios; Select the target grid parameter with the highest matching score, and based on the target grid parameter, generate a global map and at least one sub-map of the target size from the target document image.

6. The document structure extraction method according to claim 4, characterized in that, The step of inputting the global visual features and local visual features into the visual token compression module, and fusing the local visual features into the global visual features through a cross-attention mechanism to obtain compressed visual features, includes: By using a cross-attention mechanism, the global visual features are used as query vectors, and the local visual features are used as key vectors and value vectors for feature fusion, so as to fuse the local visual features into the global visual features and obtain the fused global features. Perform a convolution operation with a preset kernel size on the fused global features to obtain compressed visual features.

7. The document structure extraction method according to claim 4, characterized in that, The step of inputting the multimodal fusion features into the language decoder to generate a structured text format corresponding to the target document image includes: The multimodal fusion features are input into a pre-trained language model with a parameter size not exceeding a preset order of magnitude to generate a structured text format corresponding to the target document image; wherein, the structured text format includes heading levels, table structure, and formula format, and automatically filters headers, footers, and image elements.

8. A training device for a document structure extraction model, characterized in that, include: The dataset construction module is used to construct specialized training datasets and fine-tuning datasets. The specialized training datasets include data for various document understanding tasks, including at least two of optical character recognition, layout analysis, text localization, region text extraction, image description, and chart title generation. The fine-tuning datasets include data for document structure extraction tasks, which are used to convert document images into machine-readable structured text formats. The model building module is used to build a multimodal large model; the multimodal large model includes a shape adaptive pruning module, a visual encoder, a visual token compression module, a modal connector, and a language decoder; The pre-training module is used to pre-train the multimodal large model using the specialized training dataset to jointly learn multiple document understanding tasks. The fine-tuning module is used to fine-tune the pre-trained multimodal large model using the fine-tuning dataset, so that the multimodal large model can be adapted to the document structure extraction task, thereby obtaining a document structure extraction model.

9. An electronic device, characterized in that, The electronic device includes a processor and a memory; wherein the memory is used to store a computer program, which is loaded and executed by the processor to implement the method as claimed in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, Used to store a computer program; wherein the computer program, when executed by a processor, implements the method as described in any one of claims 1 to 7.

Images