General document structured analysis method

Abstract

The invention discloses a general document structured analysis method, and relates to the field of document intelligent analysis, and the method comprises the following steps: carrying out global layout analysis based on a thumbnail to obtain each layout area position; performing local content identification on each layout area based on the native resolution to obtain corresponding element contents; and sorting and splicing the element contents to obtain the structured document. Through decoupling global layout analysis and local content recognition, computing resources are accurately put into an information area, and unification of efficiency and precision is achieved; and multiple tasks are processed through a unified model and special elements are processed through a special algorithm, so that the integrity and accuracy of analysis are ensured.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of document intelligent parsing, and in particular to a general document structured parsing method. BACKGROUND

[0002] As a core supporting technology of digital transformation, the core goal of document parsing is to convert unstructured document images composed of high information density, complex two-dimensional spatial layout and multiple types of semantic elements into machine-readable data with complete semantics and structural fidelity. With the deepening of digital transformation in various industries, the demand for automatic processing of massive historical paper documents, electronic reports and academic literature has surged, and the precision, efficiency and general adaptability of document parsing technology have been severely challenged.

[0003] The current document intelligent parsing field faces three major contradictions, which seriously restrict the large-scale application of high-precision and high-efficiency document digitization and structuring: First, the contradiction between high-resolution input demand and computational efficiency. Fine-grained text recognition and layout analysis require native high-resolution image input, but the mainstream visual models based on Transformer architecture have a square relationship (O(N²)) between the computational complexity of their self-attention mechanism and the input sequence length. The massive image patches generated by high-resolution image segmentation cause the model's computational overhead and memory occupancy to grow exponentially, significantly reducing the inference speed and making it impossible to meet the industrial-level demand for real-time processing or large-scale batch processing.

[0004] Second, the contradiction between complex document structure understanding and semantic integrity. Modern documents often have complex structures such as multi-column layout, nested tables, cross-page charts, complex mathematical formulas and code blocks. Traditional "detection-recognition" pipeline methods decouple the task into layout detection and optical character recognition (OCR) independent modules, which easily leads to error accumulation and propagation (such as layout detection bias leading to OCR region cropping errors, which in turn causes text loss or misplacement). End-to-end multi-modal large models (Vision-Language Models, VLMs) have strong semantic understanding and sequence generation capabilities, but they have an uneven attention allocation problem in long document processing, which easily produces "hallucinations" (i.e., generating content that does not exist in the original document) or information omission, and allocates a large number of tokens to low information entropy areas (such as large areas of blank space and margins), causing serious computational redundancy.

[0005] Third, the contradiction between the high-precision analysis of specific complex elements and the limitations of general model capabilities. The analysis of mathematical formulas (including multiple lines of derivation, matrices), complex tables (including merged cells, no borders, and rotated table headers), and specific semantic blocks (such as headers, footers, references, and codes) requires simultaneous implementation of character recognition, two-dimensional spatial structure understanding, and logical relationship modeling. However, general document parsing models have issues such as structural disorder, loss of logical relationships, and misjudgment of semantic blocks, making it difficult to meet the needs of downstream tasks for high-fidelity structured data.

[0006] Currently, the mainstream technology route for document parsing is divided into two categories: Pipeline method based on traditional computer vision and OCR. This method uses modular design to divide the parsing process into multiple independent steps. Typical implementations include: PP-Structure series: PaddleOCR's document analysis system, which identifies text, tables, images, and other regions through the PP-YOLO target detection model, performs OCR recognition on text regions, and merges the results after separate structure recognition of table regions. LayoutParser: A unified toolkit that supports users to combine pre-trained layout detection models such as Detectron2 with OCR engines such as Tesseract and PaddleOCR to build a parsing pipeline. Marker: An open-source tool that integrates multiple models to achieve PDF text extraction, OCR recognition, and layout analysis, outputting results in Markdown format.

[0007] The advantages of this method are: modular design with clear logic, independent optimization of each step, and higher processing efficiency for specific tasks such as printed OCR. Its defects are: errors are propagated step by step in the pipeline, information is not shared between modules, and it is difficult to handle complex associations between elements such as cross-column text order, and it lacks robustness for complex layouts such as rotated text and irregular tables.

[0008] Method based on end-to-end multi-modal large models (VLMs). This method is based on the technology of large language models (LLMs) and builds a unified model to directly implement "image input-structured text generation". Typical implementations include: General VLMs for document applications: such as GPT-4V(ision), GeminiProVision, Qwen2-VL, etc. Treat document images as visual input and generate structured descriptions or respond to document content queries by combining prompt instructions. Document-specific large models: such as MonkeyOCR, dots.ocr, etc., pre-trained and fine-tuned through a large number of document image-text pairs, optimized for document parsing tasks, and the format restoration accuracy is better than that of general VLMs.

[0009] The advantages of this method are: end-to-end training avoids error propagation, has strong semantic understanding and context modeling capability, can effectively handle logical order and semantic association, and the overall design is simple; the defects are: high computational cost, high resolution input leads to too long sequence length, slow inference speed; when generating long structured text, it is easy to have "hallucinations" or format errors, and it is highly dependent on the quality and size of the training data.

[0010] The closest existing implementation to the technical solution of the present application is "MonkeyOCR-Pro-3B" and "dots.ocr", both of which are document-specific large models, and are committed to balancing parsing accuracy and processing efficiency, but still have the following essential defects: first, a single model is used to process a complete high-resolution image, although engineering optimization can alleviate the computational pressure, but it does not fundamentally solve the efficiency bottleneck caused by O(N²) computational complexity; second, the parsing ability of long formulas, complex tables and other fine structures is insufficient, and it is difficult to meet the demand of downstream tasks for high-fidelity structured data.

[0011] Based on the above analysis, the existing document parsing technology has three major defects: Efficiency bottleneck: the high-resolution feature extraction of the pipeline method and the long sequence processing of the end-to-end VLM are both limited by the computational complexity brought by high-resolution input, and cannot meet the industrial-level large-scale processing demand; Accuracy limitation: the pipeline method has error accumulation problem, and the end-to-end method has "hallucination" and information omission problem, both of which cannot achieve the accuracy and structure fidelity required by high-level applications in complex layout, cross-page elements and fine structure parsing; Insufficient generalization and adaptability: the existing models are all static models, and the performance upper limit is fixed after training, when facing new versions, new field documents and rare elements, there is no effective online learning and self-adaptive ability, and it needs to rely on expensive and lagging re-labeling and model iteration. SUMMARY

[0012] The present application aims to overcome the above-mentioned shortcomings, and provides an innovative document parsing method, and the specific purpose is to design a global-local decoupled process, limit the fine-grained analysis with high computational cost to information-intensive areas, thereby greatly improving the processing efficiency of high-resolution documents without losing accuracy.

[0013] The present application provides a general document structured parsing method, comprising: perform global layout analysis on the thumbnail image to obtain positions of layout regions; perform local content analysis on each layout region based on the native resolution to obtain element content; and sort and splice the element content to obtain a structured document.

[0014] In an embodiment of the present application, the global layout analysis based on the thumbnail image to obtain positions of layout regions comprises: sampling the original document image to a fixed-size thumbnail image through bilinear interpolation; inputting the thumbnail image into a visual-language model to output bounding box coordinates of all layout regions; mapping the bounding box coordinates of each layout region back to the original image coordinate system to obtain the position of each layout region in the original image.

[0015] In an embodiment of the present application, the architecture of the visual-language model comprises: a visual encoding module configured to analyze the document to obtain a visual feature map of the document; a feature fusion module configured to splice and linearly map the visual feature map in the channel dimension; and a semantic decoding module configured to convert the spliced visual feature map into a structured text sequence output.

[0016] In an embodiment of the present application, the visual-language model simultaneously predicts the position, category, rotation angle, and reading order of an element in one forward propagation.

[0017] In an embodiment of the present application, the element category comprises: basic content, including text, title, table, image, and formula; document structure, including header, footer, sidebar, page footer, and page number; specific semantics, including picture title, table title, code block, algorithm block, pinyin, and reference; logical grouping, including list box and formula box.

[0018] In an embodiment of the present application, the local content analysis based on the native resolution on each layout region to obtain element content comprises: cropping an element region from the original image according to the position of each layout region; for a layout region with a rotation angle, performing affine transformation correction; inputting the cropped and corrected image into the visual-language model again, and the visual-language model starts a corresponding analysis mode according to the category of the layout region.

[0019] In an embodiment of the present application, the visual-language model activates a corresponding parsing mode according to the category of the layout area, which includes: directly generating corresponding pure text content for text or title areas; activating atomic decomposition and reorganization for formula areas; activating OTSL-based parsing procedures for table areas; generating descriptive titles or placeholders for images; directly identifying and generating syntax-highlighted text for code blocks.

[0020] In an embodiment of the present application, the activation of atomic decomposition and reorganization for formula areas includes: determining whether the formula is a multi-line compound formula; if so, splitting it into multiple single-line atomic formulas in the vertical direction through projection analysis or models; identifying each atomic formula image to generate corresponding LaTeX code snippets; combining the LaTeX code snippets into multi-line LaTeX expressions according to the spatial position relationships between the atomic formulas.

[0021] In an embodiment of the present application, the activation of OTSL-based parsing procedures for table areas includes: performing direction correction and grid line enhancement on table layout areas; parsing the table into an OTSL sequence; converting the OTSL sequence into a standard HTML or Markdown table.

[0022] In an embodiment of the present application, it further includes training and continuously optimizing the visual-language model, and the steps include: Through visual feature clustering and metadata analysis, hierarchical sampling is performed on the original data from four dimensions of layout diversity, field diversity, element diversity, and language diversity to construct a basic corpus; Based on the basic corpus, pre-training and SFT data annotation are completed through expert collaboration and automation annotation; Apply the visual-language model to a large amount of unlabeled data and perform batch inference operations; For the same document, apply image perturbations or use different decoding parameters to allow the visual-language model to perform multiple inferences, and then compare the results of multiple inferences; For a certain element, if the output difference exceeds a certain threshold, it is marked as a difficult sample; AI pre-annotation and manual review and correction by domain experts are performed on difficult samples; Integrate the labeled data into the SFT dataset; Using the updated SFT dataset, a new round of fine-tuning training is carried out on the visual-language model.

[0023] The present application has the following beneficial effects: (1) By two-stage decoupling, high-cost native resolution calculation is only used for small-area, information-rich ROI, and global analysis is completed on a very low-resolution thumbnail. Experiments show that on the same hardware, the processing speed of the present application reaches 2.12 pages per second, which is 4 times that of MonkeyOCR-Pro-3B and 7 times that of dots.ocr, which clears the efficiency barrier for large-scale industrial deployment.

[0024] (2) The 21-element label system far exceeds the traditional method and can completely retain semantic information such as headers, footers, and code blocks, providing more abundant context for downstream RAG applications.

[0025] (3) The ADR framework makes the CDM score of complex formula recognition reach more than 90.6; the OTSL makes the TEDS score of table recognition reach the optimal on the financial statement dataset. The best results are also achieved on the reading order prediction task.

[0026] (4) In the zero-shot evaluation on the D 4 In the zero-shot evaluation on the LA, DocLayNet, etc. layout analysis dataset, the F1 score exceeds the special model (such as LayoutLMv3), proving the strong generalization ability brought by the decoupling architecture and unified multi-task training. Data augmentation and difficult sample training improve the robustness of the model to noise, deformation, and light changes.

[0027] (5) The built-in data closed-loop training process makes the system no longer a static model that is "one-time for all time". It can automatically find its own weaknesses and continuously improve itself through efficient human-machine cooperation, and can continuously adapt to new document types and formats, maintaining long-term technological leadership. BRIEF DESCRIPTION OF DRAWINGS

[0028] Figure 1 A flowchart of a general document structured analysis method in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0029] In the following description, reference is made to specific embodiments of the application. Those skilled in the art will recognize that the application can be practiced with one or more specific details of the embodiments described herein. However, it is submitted that the application should not be limited to the specific details and embodiments set forth herein.

[0030] In the present application, the embodiments are merely intended to illustrate the solutions of the present application and should not be understood as limiting.

[0031] In this specification, reference to "one embodiment" or "the embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the application. The appearances of the phrase "in one embodiment" in various places in the specification are not necessarily all referring to the same embodiment.

[0032] In addition, the numbering of the steps of the methods of the present application does not limit the order in which the method steps are performed. Unless otherwise specifically noted, the method steps can be performed in different order.

[0033] The present application is further illustrated below with reference to the accompanying drawings in connection with the following detailed description.

[0034] The core of the present application is a two-stage global-local decoupled vision-language model architecture, combined with fine-grained multi-task perception and data-driven closed-loop optimization. Its basic content includes: The first stage (global coarse resolution): uniformly scale the input document image to a lower resolution, input a lightweight vision-language model, and quickly perform multi-task layout analysis covering position, category, rotation angle, and reading order to obtain the macro "blueprint" of the document.

[0035] The second stage (local fine resolution): according to the layout "blueprint" output by the first stage, accurately crop each region of interest (ROI) from the original high-resolution image. Input these ROIs into the same or another optimized vision-language model at their native resolution (or a higher resolution that is moderately scaled but still retains details) for fine-grained content recognition and structured generation. For formulas and tables, respectively use the atomic decomposition and recombination (ADR) framework and the optimized table structure language (OTSL) for special processing.

[0036] Figure 1 A flowchart of a general document structured analysis method in an embodiment of the present application is shown.

[0037] As Figure 1As shown, in the present embodiment, the general document structure analysis method comprises: S1, global layout analysis based on thumbnail: S1.1, input preprocessing: A bilinear interpolation algorithm is used to quickly downsample the original high-resolution document image to generate a low-resolution thumbnail of a fixed size. The fixed size is a pre-set size, for example, 1036 pixels (width) x 103 pixels (height). This size configuration meets the requirement of retaining the overall layout information of the page while controlling the number of visual feature maps within a pre-set low threshold, achieving a balance between processing efficiency and information integrity.

[0038] S1.2, visual-linguistic model inference: The low-resolution thumbnail obtained by step S1.1 is input into a pre-set visual-linguistic model, and the visual-linguistic model is controlled to perform a forward propagation operation once, and the prediction results are output simultaneously, as follows: Position prediction: output the bounding box prediction results corresponding to all document elements, and the coordinate values of the bounding box are normalized values relative to the low-resolution thumbnail; Class prediction: output the semantic class prediction results corresponding to each bounding box; the semantic class is determined based on a pre-set refined label system, which contains 21 classes and is divided into the following four categories according to the functional dimension: (1) Basic content class: text, title, table, image, equation; (2) Document structure class: header, footer, aside_text, page_footnote, page_number; (3) Specific semantic class: image_caption, table_caption, code, algorithm, phonetic, reference; (4) Logical grouping class: list, equation_block (used to aggregate multi-line equations); Direction prediction: output the target rotation angle prediction results of each document element, the rotation angle is selected from the pre-set angle set {0°, 90°, 180°, 270°} and is used for angle correction of the document generated by inclined scanning; Sequential prediction: output the reading order index prediction results covering all bounding boxes, which are configured to strictly follow the human reading logic in the final output text stream, including but not limited to left-to-right, top-to-bottom reading order and cross-column coherent logical order.

[0039] In this embodiment, the vision-language model includes: Vision encoding module: NaViT (Native Resolution Vision Transformer) is used as the backbone network, with a parameter amount of 675M. The core advantage of NaViT lies in supporting dynamic resolution input and 2D-RoPE (two-dimensional rotation position encoding). Traditional ViT needs to adjust the input image to a fixed size, which will introduce distortion or loss of details. NaViT allows the model to process at a resolution close to the original aspect ratio, and 2D-RoPE can effectively inject the absolute two-dimensional coordinate information of the image block into the attention mechanism, enabling the model to better understand the spatial layout relationship of elements, which is crucial for document parsing.

[0040] Feature fusion module: In order to alleviate the pressure of long visual token sequence on the subsequent language model, this module designs a PatchMerger component. It uses the Pixel-unshuffle operation (inverse process of Pixel-shuffle) to concatenate and linearly map the channel dimension of 2x2 adjacent visual tokens (feature maps) in space, thereby reducing the spatial size by half and compressing the sequence length to 1 / 4 of the original. This operation significantly shortens the sequence length while aggregating local spatial information and preserving key visual semantics.

[0041] Semantic decoding module: Qwen2-Instruct-0.5B is used as the base language model. In order to make it better understand the multi-scale features from the vision encoder, we replace its original 1D-RoPE position encoding with M-RoPE. M-RoPE can dynamically adjust the frequency of position encoding according to the scale of input visual features, enhancing the model's perception and description ability of different scale targets in the document image. This module is responsible for converting the fused visual semantic features into structured text sequence output, such as Markdown, LaTeX or OTSL code.

[0042] S1.3, post-processing: map the predicted bounding box coordinates back to the original high-resolution image coordinate system to obtain the accurate position of each element in the original image (ROI).

[0043] S2, local content recognition based on native resolution ROI: S2.1, ROI cropping and preprocessing: According to the accurate ROI coordinates obtained in step S1.3, crop each element region from the original high-resolution image. To ensure details, the cropped image is processed at its native resolution or a set upper limit resolution. For elements with a rotation direction prediction, perform affine transformation correction first.

[0044] S2.2, Fine identification by type: Re-input the ROI image after cropping and correction into the model, which will start the matching analysis mode according to the category label corresponding to the ROI. The analysis mode corresponding to each category is as follows: Text / title area: Directly identify and generate the corresponding pure text content.

[0045] Formula area: Start the Atomic Decomposition and Recombination (ADR) process, with the following specific steps: First, determine whether the formula is a "compound formula" (i.e., a multi-line formula); If it is determined to be a compound formula, divide it into multiple "atomic formulas" (i.e., single-line formulas) along the vertical direction through simple projection analysis or a lightweight model; Perform accurate identification on each atomic formula image to generate the corresponding LaTeX code snippet; Use a structure reorganizer based on rules or a lightweight model to logically combine each LaTeX code snippet into a complete and grammatically correct multi-line LaTeX expression according to the spatial relationships between atomic formulas (such as superscripts, subscripts, fraction lines, matrix alignment symbols, etc.).

[0046] Table area: Start the OTSL-based analysis process, with the following specific steps: First, perform direction correction and grid line enhancement processing on the table class ROI image; Then parse the table into a compact Optimized Table Structure Language (OTSL) sequence, the tokens of OTSL can be directly mapped to two-dimensional grid operations of the table (such as "start row", "start column", "merge right", "merge down", "fill content: xxx", etc.), so that the length of tokens required to describe a complex table is only about 20% of the traditional HTML format; Finally, convert the OTSL sequence into a standard HTML or Markdown format table through a deterministic compiler.

[0047] Other regions: For image ROI, a descriptive title can be generated or marked as a placeholder; for code block ROI, the code content is directly identified and text with syntax highlighting is generated.

[0048] S2.3, result integration: sort and splice all element contents identified in step S2.2 according to the reading order index predicted in step S1.2, and finally assemble a complete and structured document representation.

[0049] In order to train and continuously optimize the above process, the present application designs a complete visual-language model training strategy, which specifically includes: Build balanced data base: through multi-dimensional balanced data screening, rely on visual feature clustering and metadata analysis, sample the original data from four dimensions of layout, field, element and language, and construct a balanced basic corpus to avoid the problem of model bias towards common simple documents.

[0050] Perform automated annotation application: use expert collaborative automated annotation scheme, use "expert model pipeline" instead of single model to complete pre-training and SFT data annotation. Use basic model MinerU2 to complete initial annotation, use super large scale VLM (Qwen2.5-VL-72B) to correct text semantics, and use special formula model (UniMERNet) and table model to generate high-quality LaTeX and OTSL, while ensuring annotation quality and realizing annotation automation.

[0051] Develop batch reasoning application: apply visual-language model to a large amount of unlabeled data and perform batch reasoning operation to provide data basis for subsequent difficult sample mining.

[0052] Based on the IMIC (inference consistency) strategy, the difficult samples are screened. When the model reasons on unlabeled data, the inference consistency strategy is formally started. For the same document, by adding slight image disturbance (noise, blur, slight deformation) or setting different decoding parameters, the model is driven to carry out multiple reasoning. Then compare the output results of multiple reasoning to divide the sample types: If the output of multiple reasoning of an element in the document is highly consistent, it is determined that the sample is a simple sample; If the output results of multiple reasoning are significantly different, it means that the model's decision confidence for the sample is low, and it is marked as a difficult sample.

[0053] Human-computer collaborative annotation needle: For the difficult samples excavated, carry out fine annotation work and form high-quality annotated data. The difficult samples automatically screened are included in the annotation queue, and a human-computer collaborative mode of AI pre-annotation + field expert manual review and correction is used. First, a strong model such as Gemini2.5Pro is used to complete preliminary annotation of difficult samples, and then field experts comprehensively review and correct the annotation results, and finally high-quality annotated data is obtained.

[0054] Model fine-tuning optimization: The high-quality annotated data output by the human-computer collaborative annotation is merged into the SFT dataset, and the dataset is used to fine-tune the visual-linguistic model for a new round, to specifically make up for the reasoning defects of the model on difficult samples, to improve the processing capability of the model on complex and low-confidence samples, and to complete the optimization and upgrading of the model.

[0055] The model that has completed fine-tuning optimization is re-used in actual business scenarios. Batch reasoning of unannotated data, automatic annotation and other tasks can be performed again, and in the new round of reasoning, the IMIC strategy continues to excavate new difficult samples, thereby starting the next round of "annotation-fine-tuning-application" cycle and realizing the continuous iteration and evolution of the model.

[0056] The present application proposes a complete set of core technology system and architecture scheme for document parsing, covering multiple innovative methods and special systems: The two-stage global-local decoupling document parsing method decomposes the document parsing task into global multi-task layout analysis based on low-resolution thumbnail and fine content recognition based on high-resolution local cropped image, protecting the complete process and overall architecture; The unified multi-task layout perception paradigm simultaneously predicts the document element bounding box, at least 21 types of fine semantic categories, rotation angle and global reading order in a single model, protecting the integrated technical solution; The atomic decomposition and reconstruction (ADR) mathematical formula parsing method first segments complex multi-line formulas into atomic lines, identifies each line, and then reconstructs according to the spatial structure logic, protecting the complete framework and algorithm process; The table recognition method based on optimized table structure language (OTSL) describes the table in a compact sequence language corresponding to the two-dimensional structure of the table and converts it into a standard format, protecting the technical implementation scheme; The inference consistency-based iterative mining (IMIC) strategy automatically identifies difficult samples by comparing the consistency of the model's reasoning results under slight perturbations, and applies them to the data closed-loop engine, protecting the method and actual application method; In addition, it also includes a visual-linguistic model for document parsing, which adopts a specific design combining a visual encoder supporting dynamic input and 2D-RoPE (such as NaViT) and a feature fusion module based on Pixel-unshuffle to realize sequence compression, protecting the combination scheme and overall architecture of the model.

[0057] The application has been put into practical use, and the feasibility has been fully proved through multiple rounds of experimental verification and scene application testing. 1. Performance verification: comprehensive evaluation was carried out on multiple authoritative benchmark test sets such as OmniDocBench, Ocean-OCR, olmOCR-bench, etc. The comparison models cover pipeline tools, general multi-modal large models, and special parsing large models. The application achieves optimal performance in all tasks such as text recognition, formula recognition, table recognition, and reading order extraction. In the Ocean-OCR test, the Chinese document F1 score reaches 0.965, and the overall score of olmOCR-bench is 75.2, verifying the parsing accuracy and robustness.

[0058] 2. Efficiency verification: in the OmniDocBench test set with an average of more than 1100 tokens per page, the application achieves an end-to-end throughput of 2.12 pages per second. This speed is 4 times that of MonkeyOCR-Pro-3B and 7 times that of dots.ocr. Experimental results show that the decoupled architecture proposed by the application successfully breaks through the barriers of efficiency and accuracy in high-resolution document parsing, and has the potential for large-scale industrial deployment.

[0059] 3. Scene verification: has been successfully applied to academic paper parsing, industry report structuring, historical archive digitization and other practical scenarios, and can handle complex situations such as multi-column layout, nested tables, complex formulas and rotated documents. The structured output results meet the needs of downstream RAG, intelligent analysis and other applications, and run stably and reliably.

[0060] Although the embodiments of the application are described above, it should be understood that they are presented only as examples and not as limitations. It is obvious to those skilled in the relevant art that various combinations, modifications and changes can be made without departing from the spirit and scope of the application. Therefore, the width and scope of the application disclosed herein should not be limited by the above disclosed exemplary embodiments, but should be defined only according to the appended claims and their equivalent replacements.

Claims

1. A generic document structured parsing method, characterized by, The method comprises: performing global layout analysis on a thumbnail image to obtain the positions of layout regions by a visual-language model; performing local content analysis on each layout region by the visual-language model to obtain the content of each element based on the original resolution; and sorting and splicing the element content to obtain a structured document.

2. The method of claim 1, wherein, The global layout analysis based on the thumbnail image comprises: sampling the original document image to a fixed-size thumbnail image by bilinear interpolation; inputting the thumbnail image into the visual-language model to output the bounding box coordinates of all layout regions; mapping the bounding box coordinates of each layout region back to the original image coordinate system to obtain the position of each layout region in the original image.

3. The method of claim 1, wherein, The architecture of the visual-language model comprises: a visual encoding module configured to analyze the document to obtain a visual feature map of the document; a feature fusion module configured to splice and linearly map the visual feature map in the channel dimension; and a semantic decoding module configured to convert the spliced visual feature map into a structured text sequence output.

4. The method of claim 1, wherein, The visual-language model simultaneously predicts the position, category, rotation angle, and reading order of an element in one forward propagation.

5. The method of claim 4, wherein, The element category comprises: basic content, including text, title, table, image, and formula; document structure, including header, footer, sidebar, page footer, and page number; specific semantics, including picture title, table title, code block, algorithm block, pinyin, and reference; logical grouping, including list box and formula box.

6. The method of claim 1, wherein, The local content analysis based on the original resolution comprises: cropping the corresponding element region from the original image according to the position of each layout region; for a layout region with a rotation angle, performing affine transformation correction; inputting the cropped and corrected image into the visual-language model again, and the visual-language model starts the corresponding analysis mode according to the category of the layout region.

7. The method of claim 6, wherein, The visual-language model starts the corresponding analysis mode according to the category of the layout region comprises: directly generating corresponding pure text content for a text or title region; starting atomic decomposition and recombination for a formula region; starting an OTSL-based analysis process for a table region; generating a descriptive title for an image or marking it as a placeholder; directly identifying and generating text with syntax highlighting for a code block.

8. The method of claim 7, wherein, The atomic decomposition and recombination for the formula region comprises: determining whether the formula is a multi-line compound formula; if yes, dividing the formula into multiple single-line atomic formulas in the vertical direction by projection analysis or a model; identifying each atomic formula image to generate corresponding LaTeX code snippets; combining the LaTeX code snippets into a multi-line LaTeX expression according to the spatial position relationship between the atomic formulas.

9. The method of claim 7, wherein, The OTSL-based analysis process for the table region comprises: performing direction correction and grid line enhancement on the table layout region; parsing the table into an OTSL sequence; converting the OTSL sequence into a standard HTML or Markdown table.

10. The method according to claims 1 to 9, characterized in that, The method further comprises training and continuously optimizing the visual-language model, and the steps comprise: Through visual feature clustering and metadata analysis, the original data is stratified sampled from four dimensions of layout diversity, field diversity, element diversity, and language diversity to construct a basic corpus; Based on the basic corpus, pre-training and SFT data labeling are completed through expert collaboration and automatic labeling; The visual-language model is applied to a large amount of unlabeled data to perform batch inference operations; For the same document, the visual-language model is asked to perform multiple inferences by applying image perturbations or using different decoding parameters, and then the results of multiple inferences are compared; For a certain element, if the output difference exceeds a certain threshold, it is marked as a difficult sample; The difficult samples are AI pre-labeled and manually reviewed and corrected by domain experts; The labeled data is integrated into the SFT dataset; Using the updated SFT dataset, the visual-language model is fine-tuned for a new round of training.

Images