Document analysis large model, training method and multi-task document structured analysis method

Abstract

The invention relates to a large document analysis model, a training method and a multi-task document structured analysis method, which are suitable for the fields of artificial intelligence, computer vision and natural language processing. The model comprises a visual coding module used for extracting an image feature vector from a document page image, and the extraction of the image feature vector comprises the step of capturing a dependency relationship on a spatial dimension and a channel dimension by adopting a double attention module; the visual feature projection module is used for mapping the image feature vector extracted by the visual coding module from a visual feature space to a feature space of the text decoding module to generate a visual projection feature vector; and the text decoding module is a large language model based on a Transform decoder architecture, and is used for generating document space layout information or document content corresponding to the document processing task on the document page image based on the visual projection feature vector and the token sequence after word segmentation of the cue word of the document processing task.

Description

Technical Field

[0001] This invention relates to a large-scale document parsing model, a training method, and a multi-task document structure parsing method. It is applicable to the fields of artificial intelligence, computer vision, and natural language processing. Background Technology

[0002] In the digital age, a large number of historical documents, reports, contracts, academic papers, etc., are stored and circulated in unstructured formats (such as scanned images, PDFs, and photos). These formats are easy for humans to read and print, but difficult for machines to directly understand and process, constituting a major bottleneck for information flow and knowledge mining. With the widespread use of unstructured documents such as PDFs and images in business, scientific research, and daily office work, automatically extracting, parsing, and structuring target information from them is a common pain point across industries and a key link in unlocking data value and driving business growth. Faster, more accurate, and more efficient parsing of unstructured documents into structured, editable documents is an urgent need to improve work efficiency and realize the transformation of data value.

[0003] The existing technical approaches for structured document parsing mainly include the following: (1) The method based on optical character recognition (OCR) and rule engine: The text in the document image is converted into a plain text string using OCR technology, and the position coordinates of each character or word are attached. Subsequently, the developers write a large number of heuristic rules for documents with specific layouts (such as specific types of invoices and reports). These rules infer their logical roles based on information such as text coordinates, font size, and keywords, and face the problems of poor universality and high dependence on templates.

[0004] (2) Multi-stage workflow method: Design a multi-stage workflow to realize the parsing of document structure through multiple algorithm modules such as layout analysis, regional content recognition, optical character recognition, table recognition, formula recognition, and structured reorganization. This method has the characteristics of information silos and error accumulation, and the system is highly complex and the data annotation cost is huge.

[0005] (3) Fully end-to-end approach: Train a large multi-task model and try... Figure 1 This method inputs an image once and outputs all structured results directly. While simplifying the process, it has fatal flaws: a) low computational efficiency due to the large number of model parameters, requiring activation of all parameters for each inference, resulting in low computational efficiency; b) inter-task interference, with conflicting optimization objectives for different tasks (such as text extraction and formula recognition), making it difficult for the model to converge to the optimal solution for all tasks; c) poor flexibility, making it difficult to upgrade or optimize for a specific task. Summary of the Invention

[0006] The technical problem to be solved by this invention is to provide a large-scale document parsing model, a training method, and a multi-task document structured parsing method to address the aforementioned problems.

[0007] The technical solution adopted in this invention is: a large-scale document parsing model for structured document parsing, the model comprising: The visual encoding module is used to extract image feature vectors from document page images. The extraction of image feature vectors includes using a dual attention module to capture dependencies in spatial and channel dimensions. The visual feature projection module is used to map the image feature vectors extracted by the visual encoding module from the visual feature space to the feature space of the text decoding module, thereby generating visual projection feature vectors. The text decoding module is a large language model based on the Transformer decoder architecture. It is used to generate document spatial layout information or document content on the document page image that corresponds to the document processing task, based on the visual projection feature vector and the token sequence after word segmentation of the document processing task prompt words.

[0008] The visual encoding module includes: The visual embedding module is used to segment the document page image into image blocks, and then convert the input two-dimensional image data into a one-dimensional feature vector sequence with positional information based on the position of each image block. The feature extraction module has a dual attention mechanism module, which is used to capture the dependencies of image features in the spatial and channel dimensions and generate image feature vectors.

[0009] The text decoding module includes: The word embedding and position encoding module is used to map the token to the feature space of the text decoding module and concatenate it with the visual projection feature vector to form the input feature. The input feature is injected with position information into the Transformer architecture using rotation position encoding. The decoder module includes: A normalization layer is used to ensure that the received data is at a similar scale before it enters the group query attention mechanism layer and the feedforward network layer. The group query attention mechanism layer is used to transform the input feature vector into the corresponding feature vector through three linear layers, Q, K, and V. The feedforward network layer is used to perform nonlinear transformations and feature enhancements on the contextual information extracted by the group query attention mechanism layer.

[0010] A training method for the document parsing large model employs a three-stage progressive training strategy, including: The first phase, the cross-modal alignment pre-training phase, includes: freezing the text decoding module parameters; and training using an optical character recognition dataset. The second stage, the visual-semantic fusion training stage, includes: unfreezing the text decoding module parameters; and training using a composite task dataset that includes optical character recognition and layout analysis. The third stage, the advanced cognitive ability enhancement stage, includes: freezing the parameters of the visual encoding module; and training using a comprehensive task dataset that includes optical character recognition, layout analysis, table structure recognition, and formula structure recognition. Each dataset's data unit contains a document image and a corresponding annotation file. This annotation file can simultaneously carry annotation information for character recognition, layout analysis, table structure recognition, and formula structure recognition tasks. The annotation information for layout analysis includes the logical relationships between page elements.

[0011] During the first stage of training, the loss function is minimized jointly. and This aligns the feature spaces of the visual encoder and the text encoder, while ensuring that the text recognition results are consistent with the OCR annotations. ; ; in, For image feature vectors, For positive sample text feature vectors, For negative sample text feature vectors, sim For cosine similarity, For temperature parameters, N This represents the batch sample size. Text sequences annotated for OCR, The model prediction result, where M is the length of the text sequence.

[0012] During the second stage of training, the loss function is jointly optimized. and , The loss includes bounding box regression loss. Element category loss and index loss ; ; ; ; in, middle Represents the actual bounding box coordinates. The bounding box coordinates predicted by the model. It is the intersection-union ratio of the coordinates of the two bounding boxes. It is a weighting coefficient used to balance the proportion of L1 loss and IoU loss. middle Represents the actual category label, This indicates the predicted category label. middle Indicates the sequence length. This represents the element at index t. This represents the set of all elements up to index t. That is, the input document image.

[0013] The loss function during the third stage of training is jointly optimized. and , The loss includes bounding box regression loss. Element category loss and index loss ; ; ; ; in, middle Represents the actual bounding box coordinates. The bounding box coordinates predicted by the model. It is the intersection-union ratio of the coordinates of the two bounding boxes. It is a weighting coefficient used to balance the proportion of L1 loss and IoU loss. middle Represents the actual category label, This indicates the predicted category label. middle Indicates the sequence length. This represents the element at index t. This represents the set of all elements up to index t. That is, the input document image.

[0014] A cross-modal aligned multi-task document structure parsing method includes: Obtain the input document; if the document is not in image format, render it as an image to obtain the original page image. The original page image is scaled to the target resolution and then numerically normalized to obtain a preprocessed page image. The preprocessed page image and the prompts for the page element layout analysis task are fed into the document parsing big model, and the document space layout information corresponding to the layout analysis task is output. The document space layout information is used to filter the position information of text page elements, and the image region of the page element is cropped from the preprocessed page image. The image region and the prompt words of the character recognition task are fed into the document parsing model to output the character content of the image region. The document space layout information is used to filter the position information of the page elements corresponding to the formulas, and the image region of the page element is cropped from the preprocessed page image. The image region and the prompt words of the formula structure recognition task are fed into the document parsing big model, and the formula structure content of the image region is output. The document filters the position information corresponding to the table-type page elements from the document space layout information, and crops the image region of the page element from the preprocessed page image. The image region and the prompt words of the table structure recognition task are fed into the document parsing big model, and the table structure content of the image region is output. Based on document space layout information, character content, structured formula content, and structured table content, a structured document corresponding to the input document is generated.

[0015] The target resolution is divisible by the structural parameters of the visual encoding module and is determined by preset minimum and maximum total pixel constraints.

[0016] The step of scaling the original page image to the target resolution includes: ; ; like If the pixel value is between the preset minimum pixel threshold (min_pixels) and maximum pixel threshold (max_pixels), then... , ; like ,but ; ; ; like ,but ; ; ; Where H is the target height of the original page image; W is the target width of the original page image; and P is the minimum effective granularity of information processing by the visual encoding module in the spatial dimension. For target height; The target width.

[0017] A cross-modal aligned multi-task document structure parsing apparatus includes: The format unification module is used to obtain the input document. If the document is not in an image format, it is uniformly rendered into an image to obtain the original page image. The preprocessing module is used to scale the original page image to the target resolution and perform numerical normalization on the image to obtain a preprocessed page image. The layout analysis module is used to input the preprocessed page image and the prompts for the page element layout analysis task into the document parsing big model, and output the document space layout information corresponding to the layout analysis task. The character recognition module is used to filter the position information corresponding to text page elements from the document spatial layout information, and to crop the image region of the page element from the preprocessed page image. The image region and the prompt words of the character recognition task are sent into the document parsing big model to output the character content of the image region. The formula recognition module is used to filter the position information corresponding to formula-type page elements from the document spatial layout information, and to crop the image region of the page element from the preprocessed page image. The image region and the prompt words of the formula structure recognition task are sent into the document parsing big model to output the formula structure content of the image region. The table recognition module is used to filter the position information corresponding to table-type page elements from the document spatial layout information, and to crop the image region of the page element from the preprocessed page image. The image region and the prompt words of the table structure recognition task are sent into the document parsing big model to output the table structure content of the image region. The structured document generation module is used to generate a structured document corresponding to the input document based on the document's spatial layout information, character content, structured formula content, and structured table content.

[0018] A storage medium storing a computer program executable by a processor, wherein the computer program, when executed, implements the steps of the cross-modal aligned multi-task document structure parsing method.

[0019] A multi-task document structure parsing device includes a memory and a processor. The memory stores a computer program that can be executed by the processor. When the computer program is executed, it implements the steps of the cross-modal aligned multi-task document structure parsing method.

[0020] The beneficial effects of this invention are as follows: This invention utilizes a large document parsing model to perform structured parsing of documents. This large document parsing model adopts a unified end-to-end generative model architecture, including a visual encoding module capable of processing high-resolution document images for extracting fine image features; a large language model with powerful reasoning and generation capabilities as a text decoding module; and a visual feature projection module designed between the text decoding module and the visual encoding module to realize the projection conversion from visual encoding to text encoding. The visual encoding module is based on SigLIP and optimizes the standard self-attention layer, using a dual attention module to capture dependencies in spatial and channel dimensions, thereby enabling the text decoding module to handle document processing tasks related to spatial location and document content.

[0021] The dataset used for model training in this invention covers various document types, including text-based PDFs, image-based PDFs, mobile phone photos, screenshots, and other document formats. This lays a powerful foundation for the model's general visual and layout understanding capabilities. To efficiently support this unprecedented diversity, the labeled files in the dataset can simultaneously carry annotation information for character recognition, layout analysis, table structure recognition, and formula structure recognition tasks. The layout analysis annotation information includes the logical relationships between page elements, enabling a single data sample to simultaneously drive the joint training of multiple tasks such as layout analysis, table recognition, and formula recognition. This invention semantically associates text descriptions with visual elements of drawings, enabling the model to go beyond simple layout recognition and learn deep cross-modal content understanding capabilities. This achieves a leap from "recognizing layouts" to "understanding knowledge," freeing it from dependence on specific templates.

[0022] The model training method in this invention adopts a three-stage training strategy of "modal alignment - visual-semantic fusion - advanced cognition," achieving cross-modal alignment and deep fusion of visual and semantic features. The core idea of ​​this strategy is to activate the learning capabilities of different components of the model step by step and in an orderly manner, enabling it to focus on solving different levels of sub-problems at different training stages. This effectively overcomes common problems in multimodal large models in complex document parsing tasks, such as training instability, modal alignment difficulties, and catastrophic forgetting.

[0023] To preserve the details and authenticity of the document as much as possible, this invention scales the original page image to a target resolution during preprocessing based on the original page image size, the structural parameters of the visual encoding module, and the minimum and maximum total pixel constraints. This target resolution maintains the aspect ratio of the original image and avoids content distortion, while ensuring that the adjusted size is divisible by the model's processing unit, thereby maximizing the preservation of effective information and improving processing efficiency.

[0024] This invention does not use multiple models, nor does it force a single model to complete all tasks simultaneously. Instead, it deeply reuses a single, powerful multi-task model. By dynamically constructing prompts for different document processing tasks, it precisely guides the execution of specific serialized sub-tasks (layout analysis, character recognition, formula recognition, table recognition). This simplifies the complex system that originally required multiple independent models into a unified core model. It eliminates the need to develop, train, and debug multiple independent models for each sub-task, and effectively solves the task conflict problem of end-to-end models. It achieves an optimal balance in the three dimensions of "development efficiency," "computational efficiency," and "task accuracy," which were originally difficult to achieve simultaneously, and provides a new high-performance, low-cost solution for document parsing. Attached Figure Description

[0025] Figure 1 This is a diagram of the document parsing architecture in the example.

[0026] Figure 2 This is a schematic diagram of the three-stage training strategy of "modal alignment-visual semantic fusion-advanced cognition" in the embodiment.

[0027] Figure 3 This is a flowchart of the multi-task document structured parsing method in the embodiment.

[0028] Figure 4 This is a comparison image of the input document and the parsed rendered document. The left side is the input document, and the right side is the parsed rendered result. Detailed Implementation

[0029] The embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention. The step numbers in the following embodiments are set only for ease of explanation, and there is no limitation on the order between the steps. The execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0030] In the description of this invention, "multiple" means two or more. The use of "first" and "second" is for distinguishing technical features only and should not be construed as indicating or implying relative importance, or implicitly indicating the number of indicated technical features, or the order of the indicated technical features. Furthermore, unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art.

[0031] Example 1: This example presents a cross-modal aligned multi-task document structured parsing method. It employs a large document parsing model to perform structured document parsing. This large model includes a visual encoding module, a visual feature projection module, and a text decoding module (e.g., ...). Figure 1 (As shown).

[0032] In this example, the visual encoding module includes a visual embedding module and a feature extraction module, which are used to extract image feature vectors from document page images. The extraction of image feature vectors includes using a dual attention module to capture dependencies in spatial and channel dimensions.

[0033] The visual encoding module is based on SigLIP, replacing and enhancing the standard self-attention layer in SigLIP. This standard self-attention layer is replaced by a dual attention module designed to collaboratively capture the dependencies of image features in both spatial and channel dimensions, thereby generating a more comprehensive and discriminative feature representation. It mainly consists of the following two parts: The spatial attention component is responsible for modeling long-range dependencies between different spatial locations (i.e., image patches) in an image. It computes an attention weight matrix that quantifies the correlation between any two image patches in the image, enabling the model to focus on regions relevant to the core content, regardless of their physical distance. This ensures a structured understanding of "where" in the image is important.

[0034] Channel attention component: Responsible for modeling the interdependencies between feature channels. It learns a channel weight vector to explicitly emphasize more information-rich feature channels and suppress relatively minor or redundant channels. This enables the model to adaptively select and calibrate "what" features are important for semantic understanding.

[0035] Collaborative Mechanism: By combining these two attention mechanisms into a unified module, the visual encoding module in this embodiment can perform feature selection and recalibration simultaneously and collaboratively, thereby obtaining a deeper semantic understanding of visual content.

[0036] In this embodiment, the visual embedding module converts the input two-dimensional image data into a one-dimensional feature vector sequence with positional information. The visual embedding module slides a two-dimensional convolutional layer across the input image in a non-overlapping manner, with a kernel size of P×P and a stride of P×P. Each convolutional operation covers a P×P image patch and linearly transforms it into a D-dimensional feature vector. The output tensor size is [B, D, H / P, W / P], where H / P and W / P represent the number of image patches segmented in the height and width directions, respectively. In this embodiment, P is 14, D is 1152, and the output tensor size is [1, 1152, 32, 32]. To obtain a standard sequence input, the output tensor is flattened and transposed. Flattening the last two dimensions (H / P and W / P) into a single dimension yields a tensor of size [B, D, N], where N = (H / P) × (W / P) is the total number of image patches, which in this embodiment is [1, 1152, 1024]. Transposing this tensor by swapping the last two dimensions results in another tensor of size [B, N, D], which in this embodiment is [1, 1024, 1152]. This tensor represents the image patch embedding sequence.

[0037] To enable the model to perceive the relative or absolute positions of each image patch within the original image, a learnable positional encoding vector of dimension D is needed for each image patch position (from 0 to N-1). These positional encoding vectors are injected into the image patch embedding sequence through addition operations, generating a positional encoding tensor of size [1, N, D]. This tensor is then element-wise added to the input tensor, resulting in an output tensor of size [B, N, D]. The tensor's size remains unchanged, but its values ​​contain positional information. In this embodiment, N is 1024 and D is 1152.

[0038] In this embodiment, the feature extraction module consists of L1 identical coding layers stacked together, where L1 is a configurable hyperparameter; in this example, L1=26. The internal structure of each coding layer includes: ① A dual attention mechanism module, which is jointly processed by spatial attention component and channel attention component, is used to calculate the association weights between all image patches in the sequence and update the representation of each image patch accordingly.

[0039] The spatial attention component aims to capture local spatial relationships within the feature map. It first reconstructs the input one-dimensional feature sequence [B, N, C] into a two-dimensional feature map [B, H, W, C]. For efficient computation, this module does not perform global attention computation across the entire feature map; instead, it divides it into multiple non-overlapping windows. Within each independent window, standard multi-head self-attention operations are performed, calculating the correlations between pixel positions within the window. This window-based attention mechanism significantly reduces computational complexity. After attention computation, all processed windows are recombined to restore the complete feature map, which is then further transformed and enhanced using a feedforward network.

[0040] The core objective of the channel attention component is to model the dependencies between feature channels. Unlike the spatial attention component, it does not focus on the relationships between pixel positions, but rather calculates the correlations between different feature channels. The module first maps the input feature sequence [B, N, C] into three vectors, Q, K, and V, through a linear layer. A key operation is to group the feature channel dimension (C) and reshape the tensor so that attention calculations are performed on the channel dimension within each group, while the spatial dimension (N) is treated as the feature dimension. By calculating the dot product similarity between channels, a channel attention map is generated, representing the importance and correlation of features from different channels. Finally, the V vector is weighted and summed based on this attention map to aggregate channel information, and the final feature output is completed through a feedforward network.

[0041] ② A feedforward network module, consisting of two linear layers and a GELU nonlinear activation function, is used to perform further nonlinear transformation on the output of the self-attention submodule.

[0042] ③ Two layers of normalization and residual connections are applied before and after the dual attention submodule and the feedforward network submodule, respectively, to stabilize the training process and promote information flow.

[0043] In this embodiment, the image patch embedding sequence passes through these L1 coding layers sequentially. In each layer, the feature vector of each image patch is updated according to its interaction with all other image patches in the sequence.

[0044] Input tensor size: [B, N, D], which is [1, 1024, 1152] in this embodiment. Output tensor size: [B, N, D]. Although the size of the output tensor remains unchanged, it contains richer image semantic information after deep context modeling, which is [1, 1024, 1152] in this embodiment.

[0045] After L1 encoding layers, the output feature sequence undergoes a final layer normalization process to ensure the stability of the feature distribution. Input tensor size: [B, N, D], which is [1, 1024, 1152] in this embodiment. Output tensor size: [B, N, D].

[0046] In this embodiment, the visual feature projection module is used to map the image feature vectors extracted by the visual encoding module from the visual feature space to the feature space of the text decoding module, generating a visual projection feature vector. The visual feature projection module receives image features from the visual encoding module and, through one or more linear layers, maps them into a visual projection feature vector (Feature_vision) with the same dimension as the word embedding vectors of the text decoding module.

[0047] The visual feature projection module consists of two linear layers and a GELU activation function. The input tensor has dimensions [B, N, D]. After passing through the first linear layer, it maps the feature vectors from the visual encoding module 1 from the visual feature space to the feature space of the text decoding module. Its input dimension is the size of the hidden layers in the visual model, and its output dimension is the size of the hidden layers in the language model, D. llm That is, [B, N, D] llm Following the first linear layer, a GELU non-linear activation function is applied to increase the model's non-linear expressive power. This makes the conversion from visual features to linguistic features not just a simple linear mapping, but a more complex non-linear transformation, allowing the model to learn richer correspondences. After the activation function, another linear layer is added to further refine and transform the features within the feature space of the text decoding module, outputting a size of [B, N, D]. llm In this embodiment, the D of the text decoding module llm It is 2048, meaning the output size of the visual feature projection module is [B, 1024, 2048].

[0048] In this embodiment, the text decoding module is a large-scale language model based on the Transformer decoder architecture, containing several decoder layers. It adopts an innovative group query attention mechanism and rotation position encoding, which can process long sequence context information. It is used to generate document space layout information or document content on the document page image corresponding to the document processing task based on the visual projection feature vector and the token sequence after word segmentation of prompt words for document processing tasks (layout analysis, character recognition, formula recognition, table recognition, etc.).

[0049] In this embodiment, the text decoding module uses ERNIE 4.5 as the text decoder. The input includes the visual projection features (Feature_vision) from the visual feature projection module and the token sequence after segmentation of the corresponding task prompt word (Prompt). The input size of Feature_vision is [B, N, D]. llm ].

[0050] The main structure of the text decoding module in this example is: word embedding and position encoding module, decoder layer, and layer normalization layer.

[0051] The word embedding and position encoding module is used to convert the tokens of discrete cue words into D llm A dense vector of dimensions, Feature_text. Given an input batch size of B and a token sequence length of S, the model input size is [B, S], and the output size is [B, S, D]. llm In this embodiment, it is [B, S, 2048]. Then, the Feature_vision and Feature_text vectors are concatenated to form the input feature Feature_input, with a tensor size of [B, N+S, D]. llm The input feature, Feature_input, injects positional information into the Transformer architecture using rotational positional encoding. Unlike traditional positional encoding (such as sine / cosine encoding or learnable positional embeddings), positional information is incorporated into the attention mechanism through rotational vector representation, enabling the model to naturally understand and utilize relative positional relationships.

[0052] In this embodiment, the decoder layer stacks L2 layers to gradually refine and deepen the understanding of the text. Each layer contains a layer normalization layer, a group query attention mechanism layer, and a feedforward network layer.

[0053] The normalization layer inside the decoder layer ensures that the data received by each layer is at a similar scale before entering the self-attention layer and the feedforward network layer, with input and output sizes of [B, N+S, D]. llm ].

[0054] The group query attention mechanism layer transforms the input feature vector into corresponding feature vectors through three linear layers: Q, K, and V. Each vector has a length of D. head The Q-vector input dimension is D. llm The output dimension of the n_q query headers is n_qD. head The input dimension of the K-vector is D. llm The output dimension of n_k key headers is n_kD head The input dimension of vector V is D. llmThe output dimension of n_v key headers is n_vD head In this context, n_q / n_k query headers share the same set of key and value headers. After the group attention calculation is complete, all results are concatenated and re-fused through this output projection layer, and their dimensions are transformed back to the model's standard hidden dimension D. llm .

[0055] The feedforward network layer performs non-linear transformations and feature enhancements on the contextual information extracted by the group query attention mechanism layer, including a gated projection layer, the SiLU activation function, an up-projection layer, and a down-projection layer. The gate layer reduces the dimension from D... llm Extending to higher dimensions D hidden The output of the gate layer, after passing through the SiLU activation function, is element-wise multiplied with the output of the up layer. The gate layer allows the model to dynamically control which information can flow downstream. The down layer will... hidden Reprojecting back to the model's D llm .

[0056] In this embodiment, L2=18, D head =128, n_q=16, n_k=n_v=2.

[0057] In this embodiment, the normalization layer following the decoder layer is used to perform a final normalization on the final output features of the decoder layer after all computations are completed.

[0058] like Figure 2 As shown, this embodiment employs a three-stage progressive training strategy for the document parsing large model: "modal alignment - visual-semantic fusion - advanced cognition." This strategy aims to address the technical challenges faced by existing multimodal large models when processing complex documents, such as difficulty in aligning visual and linguistic modal information, unstable training processes, and susceptibility to catastrophic forgetting. The core idea of ​​this strategy is to guide the model to systematically master the abilities from basic cross-modal alignment to deep semantic understanding, and then to complex structured information parsing through phased and focused model training. This progressive learning paradigm significantly improves training stability and the final model's performance.

[0059] Phase 1: Cross-modal Alignment Pre-training Phase. The core objective of this phase is to establish a preliminary mapping relationship between the visual and linguistic modalities, i.e., to achieve basic cross-modal alignment. Specifically, it aims to enable the model to align the visual features extracted from an image with the corresponding textual semantic features of that image region in the same feature space. In this phase, a large-scale optical character recognition dataset is primarily used for training. To ensure training stability and efficiency, the parameters of the text decoding module are frozen, preventing it from participating in gradient updates; only the connection between the visual encoding module and the visual feature projection module are trained.

[0060] At this stage, the loss function is minimized jointly. and This aligns the feature spaces of the visual encoder and the text encoder, while ensuring that the text recognition results are consistent with the OCR annotations. and The loss functions are as follows: ; ; in, For image feature vectors, For positive sample text feature vectors, For negative sample text feature vectors, sim Here, represents the cosine similarity, and represents the temperature parameter. N This represents the batch sample size. Text sequences annotated for OCR, The model prediction result, where M is the length of the text sequence.

[0061] Phase Two: Visual-Semantic Fusion Training. Building upon the basic modal alignment already achieved, the goal of this phase is to enable the model to learn the document's layout information and achieve a deep fusion of visual structure and textual semantics. The model needs to understand that text is not merely a sequence of characters; its visual attributes on the page, such as position, size, and font, also contain rich semantic information (e.g., headings, paragraphs, lists). In this phase, a composite task dataset including optical character recognition and layout analysis is used for training. This dataset not only provides document content but also annotates the bounding box coordinates, category information, and semantic indexes of page elements. In this phase, the visual encoding module, visual feature projection module, and text decoding module are unfrozen and fine-tuned. The visual encoding module needs to learn feature extraction capabilities better suited to document layout analysis; the visual feature projection module needs further optimization of its cross-modal mapping capabilities; and the text decoding module needs to learn to generate structured descriptions or answers based on features fused with layout information.

[0062] Joint optimization during this training phase and , The loss includes bounding box regression loss. Element category loss and index loss ; ; ; ; Phase Three: Advanced Cognitive Ability Enhancement Phase. This phase focuses on training the model to parse highly structured complex elements such as tables and formulas, while preventing "catastrophic forgetting" of learned general layout knowledge. A comprehensive task dataset with multi-task annotations, including optical character recognition, layout analysis, table structuring, and formula structuring, is used. In this phase, the parameters of the visual encoding module are frozen, and only the visual feature projection module and the text decoding module are trained. Freezing the visual encoding module is to preserve the general layout understanding ability learned in the first two phases and prevent the model from experiencing "catastrophic forgetting" when learning specific, fine structures such as tables and formulas—that is, losing the ability to recognize basic layout elements such as ordinary paragraphs and headings. The training focuses on enabling the language decoder to learn to interpret the complex structural features extracted by the visual encoder and convert them into structured output. The loss function in Phase Three remains consistent with that in Phase Two.

[0063] The dataset construction methods for the optical character recognition dataset in the first stage, the composite task dataset in the second stage, and the integrated task dataset in the third stage in this embodiment include: I. Data Source Collection. To ensure broad coverage of the dataset, data sources include, but are not limited to: publicly available academic literature databases, scanned book archives, corporate financial reports, publicly available government documents and standards, internet web archives, and various invoices, notes, exam papers, patent documents, etc., collected through crowdsourcing or automation. The collected document formats cover text-based PDFs, image-based PDFs, and image files obtained through high-resolution scanning or mobile device photography (e.g., JPEG, PNG, TIFF formats), as well as common screenshots to simulate various real-world document acquisition scenarios. Document types include books, academic papers (single-column and multi-column), notes, exam papers, PPTs, newspapers, magazines, drawings, financial reports, textbooks, standards, web pages, industry reports, patents, official documents, invoices, and more than ten other real-world document types.

[0064] II. Data Annotation. The collected document images are annotated to generate annotation files corresponding to the document images. The document image and its corresponding annotation file constitute a data unit of the dataset.

[0065] In this embodiment, the annotation file can simultaneously carry annotation information for character recognition, layout analysis, table structure recognition, and formula structure recognition tasks. The annotation information for layout analysis includes the logical relationships between page elements. To address the diversity of annotation files, this example proposes a unified and scalable JSON data structure. This structure embeds dedicated structured supervision information (such as HTML and LaTeX) into different page elements like tables and formulas, enabling a single data sample to simultaneously drive the joint training of multiple tasks, including layout analysis, table recognition, and formula recognition.

[0066] In this embodiment, a top-level JSON object represents a document page or a standalone document image. This object contains the following key fields: Page metadata: Describes the page's global attributes: image_path: Points to the path of the corresponding document image file; image_size: The width and height of the image, such as {"width": 2480, "height":3508}; doc_type: The document type tag, such as "academic_paper", "financial_report", "patent", "textbook", "invoice", etc. source_format: The format of the original document, such as "image_pdf" (image PDF), "text_pdf" (text PDF), "photo" (image), "screenshot" (screenshot).

[0067] Page Element List: Each page element is an object representing an independent layout unit on the page (such as a text block, heading, table, image, formula, etc.). Each layout unit object contains the following fields, collectively known as the layout analysis dataset: element_id: A unique identifier for each element within the page; bounding_box: The coordinates of the element's position on the page, defining the element's rectangular bounding box, such as [x_min, y_min, x_max, y_max]. category: Category tags for page elements, used for layout detection tasks. This tag set includes, but is not limited to: "paragraph", "title", "table", "formula", "figure", "list", "header_footer", "image_caption", etc. order_index: Indicates the position of this element in the logical reading order of the document.

[0068] To support multi-task training, this data structure embeds task-specific detailed annotation information within page elements of specific categories: For elements with categories such as "paragraph", "title", or "list", add a "text_content" field to contain the accurate document content within that text block after OCR recognition and proofreading.

[0069] For elements with the category "table", an "html" and "otsl" encoded representation of the table structure and content is added to provide structured supervision information for the table structure recognition task.

[0070] For elements with the category "formula", an "latex" field is added to describe the formula content and structure, providing accurate supervision information for formula recognition tasks.

[0071] To further expand the depth and breadth of the dataset to highly complex and specialized engineering and technical fields, this embodiment specifically introduces unique knowledge documents from fields including but not limited to construction engineering, hydropower engineering, wind power engineering, road engineering, bridge engineering, tunnel engineering, railway engineering, ecological restoration engineering, electromechanical engineering, surveying and mapping engineering, and geological engineering. These documents include large-scale, extremely complex engineering drawings, technical specifications, as-built reports, bills of materials, and construction plans as core data sources. These documents are characterized by high text-image coupling, extremely high information density, and the presence of numerous non-natural language symbols and illustrations. Through refined annotation of these documents, the dataset of this invention can train models to understand the deep semantic relationships between textual descriptions (such as "A-3 load-bearing beam, using C50 concrete") and specific visual elements in drawings (such as the structure marked "A-3" on the drawing), achieving a leap from "layout recognition" to "engineering semantic understanding." This is a capability that general document datasets cannot achieve.

[0072] This embodiment constructs a dataset with unprecedented diversity in document type, source format, and task dimension, enabling the model to learn universal document visual and structural features during the pre-training or fine-tuning stage, thereby gaining general document understanding capabilities. This solves the technical problems in existing technologies, such as limited training dataset size, single task, and insufficient document type coverage, which lead to weak model generalization ability and reliance on specific templates.

[0073] like Figure 3 As shown, the cross-modal aligned multi-task document structured parsing method in this embodiment specifically includes the following steps: S1. Obtain the input document. If the document is not in image format, render it as an image to obtain the original page image.

[0074] In this embodiment, the received document input can be in various formats, including but not limited to native digital documents (such as text-based PDFs and image-based PDFs) or existing screenshots and collected page images (such as JPEG, PNG, and TIFF formats). If the input is a non-image format document such as a PDF, a high-fidelity rendering engine is used to convert each page of the document into a digital image. During this process, a relatively high rendering resolution (such as 300 DPI) is set to ensure that the image contains sufficiently rich detail information, especially for tiny characters, complex formulas, and fine table lines, thereby laying a high-quality data foundation for subsequent recognition steps. Each page image output in this step is called the original page image.

[0075] S2. Scale the original page image Image to the target resolution and normalize the image values ​​to obtain the preprocessed page image image_pre.

[0076] This step preprocesses the original page image (Image) to adapt it to the input requirements of the large document parsing model, improving the accuracy and stability of the model's perception. First, the dimensions (height H and width W) of the original page image are obtained. Unlike traditional methods that use fixed-size scaling, this embodiment employs a dynamic adaptive resolution algorithm to preserve as much information and proportion as possible in the original image, avoiding severe distortion of image content due to stretching, and meeting the requirements of subsequent model input.

[0077] In this example, the dynamic adaptive resolution algorithm calculates an optimal target resolution based on the structural parameters of the visual encoder within the large document parsing model, combined with preset minimum and maximum total pixel constraints. , The target resolution maintains the original image aspect ratio and avoids content distortion while ensuring that the adjusted size is divisible by the model's processing units, thereby maximizing the retention of effective information and improving processing efficiency. The image with adjusted resolution is then numerically normalized, including linearly scaling its pixel values ​​to the [0, 1] range and standardizing them using predefined mean and standard deviation. The output is the preprocessed page image, image_pre.

[0078] S2.1: Receive the input image. To ensure the consistency of the image color channels, check and convert all non-RGB format images (such as RGBA, L, etc.) to three-channel RGB format. Subsequently, convert the image data format to array format for subsequent numerical calculations and operations.

[0079] S2.2: Extract the original height H and width W of the input image.

[0080] S2.3: Set the minimum number of pixels (min_pixels) and the maximum number of pixels (max_pixels), in the visual embedding module M. 11 In this context, P represents the lower limit of image resolution, min_pixels represents the upper limit of image resolution, and P represents the minimum effective granularity of information processed by the model in the spatial dimension.

[0081] S2.4: Calculate the optimal target height based on the following constraints. and target width : A. Integer Divisibility Constraint: Ensures that the adjusted... and All can be embedded by the visual module M 11 Divisible by P in the equation.

[0082] ; ; B. Total Pixel Count Constraint: Ensure the adjusted total number of pixels. × It lies between the preset minimum pixel threshold (min_pixels) and maximum pixel threshold (max_pixels). If the requirements are met, then... = , ;if Adjust using the following formula: ; ; ; if Then adjust using the following formula: ; ; ; C. Aspect Ratio Preservation: Under the premise of satisfying the above constraints, maintain the aspect ratio of the original image as much as possible to minimize the stretching or compression distortion of the image content.

[0083] S2.5: Using a preset interpolation algorithm, the width and height of the image are precisely scaled from (H, W) to the calculated target resolution. ).

[0084] S2.6: Value range scaling of the image array. Linearly maps the pixel values ​​of the image from their original range to a normalized floating-point range, such as [0, 1), by multiplying by a scaling factor.

[0085] S2.7: Standardization of image array distribution: Using predefined channel mean vectors and standard deviation vectors, the images are standardized to make the data distribution more consistent with the statistical characteristics during model training, which helps to accelerate model convergence and improve performance.

[0086] S3. Input the preprocessed page image and the prompts for the page element layout analysis task into the document parsing model, and output the document space layout information corresponding to the layout analysis task.

[0087] In this embodiment, the prompts for the page element layout analysis task explicitly instruct the model to perform the document structure analysis task, such as: "Please detect all layout elements on this page and list their categories and positions in semantic order." The preprocessed page image and the prompts for the page element layout analysis task are fed into the document parsing big model. In one forward propagation, two tasks are jointly completed: identifying all predefined page element categories in the page and outputting the precise bounding box coordinates of each element instance.

[0088] In this embodiment, the predefined page element categories include 13 types: titles, text paragraphs, images, legends, tables, table titles, table footnotes, formulas, formula numbers, page footnotes, lists, page numbers, headers and footers, etc. The identified page elements are semantically ordered. This ordering is not limited to a simple top-to-bottom or left-to-right geometric sequence; model M can understand the logical relationships between elements (e.g., legends follow images, and table titles are above tables), and assigns a semantic order index to each detected element instance.

[0089] In this example, the document parsing model outputs a structured detection result for the layout analysis task: a structured list of document spatial layout information. Each item in the list contains: Category information: One of the 13 categories mentioned above to which this page element belongs; Bounding box coordinates (bbox): a quadruple (x_min, y_min, x_max, y_max) that precisely represents the position and extent of the element in the preprocessed image Image_pre; Semantic index order_index: The semantic order of page elements in the document as understood by the model.

[0090] S4. From the document space layout information generated in step S3, filter the position information (boundary box coordinates bbox) corresponding to text-type page elements (titles, text paragraphs, legends, table titles, table footnotes, formula numbers, page footnotes, lists, page numbers, headers and footers). Based on the position information, crop the image regions of the text-type page elements from the preprocessed page image Image_pre to obtain a series of small image slices crop_text_i. Pair each image slice crop_text_i with the prompt word Prompt_text for the character recognition task, and input them sequentially into the document parsing model for inference. The document parsing model outputs the corresponding high-precision character content text_i. In this embodiment, the prompt word Prompt_text for the character recognition task is "Please recognize all the text in the image".

[0091] S5. From the document space layout information generated in step S3, filter the position information corresponding to the formula-type page elements, and crop the image regions of the formula-type page elements from the preprocessed page image Image_pre according to the position information, obtaining a series of small image slices crop_formula_j. Pair each image slice crop_formula_j with the prompt word Prompt_formula for the formula structure recognition task, and input them sequentially into the document parsing model for inference. The document parsing model outputs the corresponding formula structure content, such as a LaTeX string. In this embodiment, the prompt word Prompt_formula for the formula structure recognition task is "Please convert the mathematical formula in the image into LaTeX code".

[0092] S6. From the document spatial layout information generated in step S3, filter the position information corresponding to table-type page elements, and crop the image regions of table-type page elements from the preprocessed page image Image_pre according to the position information, obtaining a series of small image slices crop_table_k. Pair each image slice crop_table_k with the prompt word Prompt_table for the table structure recognition task, and input them sequentially into the document parsing model for inference. The document parsing model outputs the corresponding table structure content, such as HTML encoding, LaTeX encoding, or OTSL encoding. In this embodiment, the prompt word Prompt_table for the table structure recognition task is "Please parse the table in the image into OTSL encoding". The table recognition result uses OTSL encoding. After obtaining the OTSL encoding, it is converted into an HTML-encoded table according to the encoding conversion rules.

[0093] S7. Based on the document space layout information generated in step S3, merge the character content obtained in S4, the structured formula content obtained in S5, and the structured table content obtained in S6 to generate a structured document corresponding to the input document.

[0094] This embodiment strictly follows the traversal of the structured page object list generated in step S3, which is ordered semantically, and iterates through each detected element. When an element category is encountered, the corresponding identified content (text_i, latex_j, or table_data_k) is found from the results of S4, S5, or S6. For non-text elements such as images, their bounding box information or image slices are recorded. The category, bounding box coordinates, semantic order index, and the specific content identified (text, LaTeX, HTML table, etc.) of each element are populated into the final data structure. This result not only completely preserves the entire content of the original document but also accurately records its layout and semantic structure information.

[0095] Figure 4 This is a comparison image of the input document and the rendered document after being parsed using the structured parsing method of this embodiment. The left side is the input document, and the right side is the rendered result after parsing.

[0096] Example 2: This example is a cross-modal aligned multi-task document structure parsing device, comprising: The format unification module is used to obtain the input document. If the document is not in an image format, it is uniformly rendered into an image to obtain the original page image. The preprocessing module is used to scale the original page image to the target resolution and perform numerical normalization on the image to obtain a preprocessed page image. The layout analysis module is used to input the preprocessed page image and the prompts for the page element layout analysis task into the document parsing big model, and output the document space layout information corresponding to the layout analysis task. The character recognition module is used to filter the position information corresponding to text page elements from the document spatial layout information, and to crop the image region of the page element from the preprocessed page image. The image region and the prompt words of the character recognition task are sent into the document parsing big model to output the character content of the image region. The formula recognition module is used to filter the position information corresponding to formula-type page elements from the document spatial layout information, and to crop the image region of the page element from the preprocessed page image. The image region and the prompt words of the formula structure recognition task are sent into the document parsing big model to output the formula structure content of the image region. The table recognition module is used to filter the position information corresponding to table-type page elements from the document spatial layout information, and to crop the image region of the page element from the preprocessed page image. The image region and the prompt words of the table structure recognition task are sent into the document parsing big model to output the table structure content of the image region. The structured document generation module is used to generate a structured document corresponding to the input document based on the document's spatial layout information, character content, structured formula content, and structured table content.

[0097] Example 3: This example is a storage medium that stores a computer program that can be executed by a processor. When the computer program is executed, it implements the steps of the cross-modal aligned multi-task document structure parsing method.

[0098] Example 4: This example is a multi-task document structure parsing device, which has a memory and a processor. The memory stores a computer program that can be executed by the processor. When the computer program is executed, it implements the steps of the cross-modal aligned multi-task document structure parsing method.

[0099] Furthermore, although the invention has been described in the context of functional modules, it should be understood that, unless otherwise stated, one or more of the aforementioned functions and / or features may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the invention. Rather, given the properties, functions, and internal relationships of the various functional modules in the apparatus disclosed herein, the actual implementation of the module will be understood within the scope of conventional skill of an engineer. Therefore, those skilled in the art can implement the invention as set forth in the claims using ordinary techniques without excessive experimentation. It is also understood that the specific concepts disclosed are merely illustrative and not intended to limit the scope of the invention, which is determined by the full scope of the appended claims and their equivalents.

[0100] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0101] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-including system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0102] More specific examples (a non-exhaustive list) of computer-readable media include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the aforementioned program can be printed, because the aforementioned program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0103] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0104] In the foregoing description of this specification, references to terms such as "one embodiment," "another embodiment," or "some embodiments" indicate that a specific feature, structure, material, or characteristic described in connection with an embodiment or example is included in at least one embodiment or example of the present invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0105] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

[0106] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.

Claims

1. A large-scale document parsing model for structured document parsing, characterized in that, The model includes: The visual encoding module is used to extract image feature vectors from document page images. The extraction of image feature vectors includes using a dual attention module to capture dependencies in spatial and channel dimensions. The visual feature projection module is used to map the image feature vectors extracted by the visual encoding module from the visual feature space to the feature space of the text decoding module, thereby generating visual projection feature vectors. The text decoding module is a large language model based on the Transformer decoder architecture. It is used to generate document spatial layout information or document content on the document page image that corresponds to the document processing task, based on the visual projection feature vector and the token sequence after word segmentation of the document processing task prompt words.

2. The document parsing large model according to claim 1, characterized in that, The visual encoding module includes: The visual embedding module is used to segment the document page image into image blocks, and then convert the input two-dimensional image data into a one-dimensional feature vector sequence with positional information based on the position of each image block. The feature extraction module has a dual attention mechanism module, which is used to capture the dependencies of image features in the spatial and channel dimensions and generate image feature vectors.

3. The document parsing large model according to claim 1, characterized in that, The text decoding module includes: The word embedding and position encoding module is used to map the token to the feature space of the text decoding module and concatenate it with the visual projection feature vector to form the input feature. The input feature is injected with position information into the Transformer architecture using rotation position encoding. The decoder module includes: A normalization layer is used to ensure that the received data is at a similar scale before it enters the group query attention mechanism layer and the feedforward network layer. The group query attention mechanism layer is used to transform the input feature vector into the corresponding feature vector through three linear layers, Q, K, and V. The feedforward network layer is used to perform nonlinear transformations and feature enhancements on the contextual information extracted by the group query attention mechanism layer.

4. A training method for a large document parsing model as described in any one of claims 1 to 3, characterized in that, A three-stage progressive training strategy is adopted, including: The first phase, the cross-modal alignment pre-training phase, includes: freezing the text decoding module parameters; and training using an optical character recognition dataset. The second stage, the visual-semantic fusion training stage, includes: unfreezing the text decoding module parameters; and training using a composite task dataset that includes optical character recognition and layout analysis. The third stage, the advanced cognitive ability enhancement stage, includes: freezing the parameters of the visual encoding module; and training using a comprehensive task dataset that includes optical character recognition, layout analysis, table structure recognition, and formula structure recognition. Each dataset's data unit contains a document image and a corresponding annotation file. This annotation file can simultaneously carry annotation information for character recognition, layout analysis, table structure recognition, and formula structure recognition tasks. The annotation information for layout analysis includes the logical relationships between page elements.

5. The training method for a large document parsing model according to claim 4, characterized in that, During the first stage of training, the loss function is minimized jointly. and This aligns the feature spaces of the visual encoder and the text encoder, while ensuring that the text recognition results are consistent with the OCR annotations. ； ； in, For image feature vectors, For positive sample text feature vectors, Here, is the feature vector of the negative sample text, and sim is the cosine similarity. Here, N is the temperature parameter, and N is the batch sample size. Text sequences annotated for OCR, The model prediction result, where M is the length of the text sequence.

6. The training method for a large document parsing model according to claim 4, characterized in that, During the second stage of training, the loss function is jointly optimized. and , The loss includes bounding box regression loss. Element category loss and index loss ; ； ； ； in, middle Represents the actual bounding box coordinates. The bounding box coordinates predicted by the model. It is the intersection-union ratio of the coordinates of the two bounding boxes. It is a weighting coefficient used to balance the proportion of L1 loss and IoU loss. middle Represents the actual category label, This indicates the predicted category label. middle Indicates the sequence length. This represents the element at index t. This represents the set of all elements up to index t. That is, the input document image.

7. The training method for a large document parsing model according to claim 4, characterized in that, The loss function during the third stage of training is jointly optimized. and , The loss includes bounding box regression loss. Element category loss and index loss ; ； ； ； in, middle Represents the actual bounding box coordinates. The bounding box coordinates predicted by the model. It is the intersection-union ratio of the coordinates of the two bounding boxes. It is a weighting coefficient used to balance the proportion of L1 loss and IoU loss. middle Represents the actual category label, This indicates the predicted category label. middle Indicates the sequence length. This represents the element at index t. This represents the set of all elements up to index t. That is, the input document image.

8. A cross-modal aligned multi-task document structured parsing method, characterized in that, include: Obtain the input document; if the document is not in image format, render it as an image to obtain the original page image. The original page image is scaled to the target resolution and then numerically normalized to obtain a preprocessed page image. The preprocessed page image and the prompts for the page element layout analysis task are fed into the document parsing big model, and the document space layout information corresponding to the layout analysis task is output. The document space layout information is used to filter the position information of text page elements, and the image region of the page element is cropped from the preprocessed page image. The image region and the prompt words of the character recognition task are fed into the document parsing model to output the character content of the image region. The document space layout information is used to filter the position information of the page elements corresponding to the formulas, and the image region of the page element is cropped from the preprocessed page image. The image region and the prompt words of the formula structure recognition task are fed into the document parsing big model to output the formula structure content of the image region. The document filters the position information corresponding to the table-type page elements from the document space layout information, and crops the image region of the page element from the preprocessed page image. The image region and the prompt words of the table structure recognition task are fed into the document parsing big model, and the table structure content of the image region is output. Based on document space layout information, character content, structured formula content, and structured table content, a structured document corresponding to the input document is generated.

9. The cross-modal aligned multi-task document structured parsing method according to claim 8, characterized in that, The target resolution is divisible by the structural parameters of the visual encoding module and is determined by preset minimum and maximum total pixel constraints.

10. The cross-modal aligned multi-task document structured parsing method according to claim 9, characterized in that, The step of scaling the original page image to the target resolution includes: ； ； like × If the pixel value is between the preset minimum pixel threshold (min_pixels) and maximum pixel threshold (max_pixels), then... = , ; like ,but ； ； ； like ,but ； ； ； Where H is the target height of the original page image; W is the target width of the original page image; and P is the minimum effective granularity of information processing by the visual encoding module in the spatial dimension. For target height; The target width.

11. A cross-modal aligned multi-task document structure parsing device, characterized in that, include: The format unification module is used to obtain the input document. If the document is not in an image format, it is uniformly rendered into an image to obtain the original page image. The preprocessing module is used to scale the original page image to the target resolution and perform numerical normalization on the image to obtain a preprocessed page image. The layout analysis module is used to input the preprocessed page image and the prompts for the page element layout analysis task into the document parsing big model, and output the document space layout information corresponding to the layout analysis task. The character recognition module is used to filter the position information corresponding to text page elements from the document spatial layout information, and to crop the image region of the page element from the preprocessed page image. The image region and the prompt words of the character recognition task are sent into the document parsing big model to output the character content of the image region. The formula recognition module is used to filter the position information corresponding to formula-type page elements from the document spatial layout information, and to crop the image region of the page element from the preprocessed page image. The image region and the prompt words of the formula structure recognition task are sent into the document parsing big model to output the formula structure content of the image region. The table recognition module is used to filter the position information corresponding to table-type page elements from the document spatial layout information, and to crop the image region of the page element from the preprocessed page image. The image region and the prompt words of the table structure recognition task are sent into the document parsing big model to output the table structure content of the image region. The structured document generation module is used to generate a structured document corresponding to the input document based on the document's spatial layout information, character content, structured formula content, and structured table content.

12. A storage medium having a computer program stored thereon that can be executed by a processor, characterized in that, When the computer program is executed, it implements the steps of the cross-modal aligned multi-task document structure parsing method according to any one of claims 8 to 10.

13. A multi-task document structure parsing device, comprising a memory and a processor, wherein the memory stores a computer program executable by the processor, characterized in that, When the computer program is executed, it implements the steps of the cross-modal aligned multi-task document structure parsing method according to any one of claims 8 to 10.

Images