A knowledge question and answer method based on a multi-modal document understanding large model of a graph structure
By using a graph-based multimodal document understanding model, the problem of insufficient parsing ability of existing document understanding models for complex layouts and novel page layouts is solved, achieving higher accuracy and robustness, and simplifying system deployment.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- 北京中科闻歌科技股份有限公司
- Filing Date
- 2026-03-06
- Publication Date
- 2026-06-05
AI Technical Summary
Existing multimodal document understanding models suffer from performance degradation in complex layouts and novel document formats due to insufficient OCR accuracy, low efficiency, and difficulty in integrating document structure information.
A large-scale multimodal document understanding model based on graph structure is adopted. By generating heterogeneous document graphs and utilizing graph bias attention mechanism, the explicit graph structure information of the document is natively integrated into the attention calculation process of Transformer, realizing end-to-end multimodal information fusion and reasoning.
It enhances the model's ability to analyze complex layouts and fuse multimodal information, possesses excellent generalization ability, achieves higher accuracy and robustness, and simplifies system deployment and application.
Smart Images
Figure CN122152998A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of knowledge question answering technology, and in particular to a knowledge question answering method for a large-scale multimodal document understanding model based on graph structure. Background Technology
[0002] In the field of multimodal document intelligence, existing technical solutions mainly evolve around two major technical paradigms, aiming to accurately extract information and answer questions from complex document images. The first is the cascaded paradigm of "OCR + language model," which employs a step-by-step processing flow, typically represented by the LayoutLM series of models. It first uses independent optical character recognition (OCR) engines (such as Tesseract and PaddleOCR) to preprocess the document image, extracting the text content and its spatial layout information, such as its two-dimensional coordinate bounding box on the page. Then, these text and layout features are fed as input into a large language model based on the Transformer architecture (such as a BERT variant) for deep semantic understanding and reasoning. While this approach achieves preliminary joint modeling of text and layout by explicitly introducing coordinate encoding and has made progress in several early benchmark tests, it has inherent fundamental flaws: First, the model performance is highly dependent on the accuracy of the preceding OCR module. Any missed detections, false detections, or coordinate errors in the OCR process will directly transmit and impair the performance of downstream understanding tasks, resulting in error accumulation. Second, this two-stage pipeline design leads to low system efficiency and makes it difficult to handle scenarios that OCR technology itself is not good at, such as blurred images, complex handwriting, mathematical formulas, and pages with tightly interwoven text and images.
[0003] To overcome the drawbacks of the cascaded paradigm, the research community has turned to exploring a unified "end-to-end" model paradigm. These models aim to eliminate explicit reliance on OCR, directly inputting the original document image into a unified neural network architecture to simultaneously learn visual features and semantic understanding in an end-to-end manner. A representative work is Donut, which uses a visual encoder (such as SwinTransformer) to extract image features, and then directly generates the target text using a sequence-to-sequence decoder (such as BART). With the development of multimodal large-scale models, more research, such as DocLLM and UReader, attempts to use large language models as the cognitive core, directly aligning and fusing them with visual features. However, to adapt to the Transformer's self-attention mechanism, these mainstream end-to-end models typically need to "flatten" the visual tokens after image segmentation into a one-dimensional sequence. This process inevitably disrupts the inherent two-dimensional spatial relationships and hierarchical structural information of the document (such as chapter nesting, table row and column relationships, and the correspondence between charts and explanatory text). Meanwhile, compressing all visual and potential textual information of the entire document into a long sequence easily leads to model attention being scattered, making it difficult to focus on key regions and their structured relationships. Although some studies have attempted to introduce graph neural networks to explicitly model document structure, these methods often employ loose coupling or post-fusion approaches, failing to deeply and natively integrate the rich relational information represented by the graph structure into the core attention computation and inference path of the Transformer. This architectural limitation makes it difficult for existing models to learn general, transferable prior knowledge of document structure. When faced with documents outside the training data distribution that have novel or complex layouts, their understanding and inference performance will significantly degrade, failing to achieve truly robust and general document intelligent understanding capabilities. Summary of the Invention
[0004] To address the aforementioned technical problems, this invention provides a knowledge question answering method for a large-scale multimodal document understanding model based on graph structure. By adopting a graph structure-aware modeling paradigm, it fundamentally improves the multimodal document understanding model's ability to parse complex layouts, fuse multimodal information, and generalize to unknown formats, ultimately achieving superior performance in tasks such as document knowledge question answering.
[0005] According to a first aspect of the present invention, a knowledge question answering method for a large-scale multimodal document understanding model based on graph structure is provided, the method comprising the following steps:
[0006] S1, Obtain the target task and the target document graph corresponding to the target task.
[0007] S2, Generate a target heterogeneous document graph corresponding to the target document graph based on the target document graph.
[0008] S3, input the target task and the target heterogeneous document graph into a preset large model to obtain the target question and answer text corresponding to the target task.
[0009] This invention has at least the following beneficial effects: a knowledge question answering method based on a graph-structured multimodal document understanding large model, comprising: S1, obtaining a target task and a target document graph corresponding to the target task; S2, generating a target heterogeneous document graph corresponding to the target document graph based on the target document graph; S3, inputting the target task and the target heterogeneous document graph into a preset large model to obtain the target question answer text corresponding to the target task; achieving higher accuracy and robustness on multiple public benchmark tasks (such as document question answering, graph question answering, and table understanding); exhibiting excellent generalization ability for novel and complex document layouts; and achieving simpler and more efficient system deployment and application through an end-to-end unified architecture. Attached Figure Description
[0010] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0011] Figure 1 The flowchart illustrates a knowledge-based question-answering method for a large-scale multimodal document understanding model based on graph structure, provided in an embodiment of the present invention. Detailed Implementation
[0012] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0013] This invention provides a knowledge question answering method for a large-scale multimodal document understanding model based on graph structures, such as... Figure 1 As shown, the method includes the following steps:
[0014] S1, Obtain the target task and the target document graph corresponding to the target task.
[0015] Specifically, the target document image is
[0016] S2, Generate a target heterogeneous document graph corresponding to the target document graph based on the target document graph.
[0017] Furthermore, step S2 also includes the following steps:
[0018] S21, perform node processing on the target document graph to obtain a target heterogeneous node set of the target document graph, wherein the target heterogeneous node set includes a first heterogeneous node set and a second heterogeneous node set.
[0019] Furthermore, the first heterogeneous node set includes N1 first heterogeneous nodes, wherein the first heterogeneous nodes are target text features extracted from the target document graph through a preset OCR model.
[0020] Furthermore, the second heterogeneous node set includes N2 second heterogeneous nodes, wherein the second heterogeneous nodes are non-textual features of the target document graph extracted by the preset target detection model, such as images, table areas, charts, and components of charts in the target document graph, and the components of charts include bars, coordinate axes, and legends.
[0021] Preferably, the preset target detection model is a preset YOLO or a preset Faster R-CNN.
[0022] S22, perform edge processing on the target document graph to obtain a target heterogeneous edge set of the target document graph, wherein the target heterogeneous edge set includes N3 target heterogeneous edges, and each target heterogeneous edge is an edge between any two target heterogeneous nodes, wherein the target heterogeneous node is either a first heterogeneous node or a second heterogeneous node. Further, the target heterogeneous edge represents the association relationship between any two target heterogeneous nodes. For example, the association relationship includes, but is not limited to:
[0023] Spatial adjacency refers to the relationship between two nodes that are physically close to each other.
[0024] Logical order relation (in English: logical_next) represents the relationship between consecutive text features that are connected in the order of reading.
[0025] Row-column membership (also known as row_membership or col_membership) refers to the relationship between all cell nodes located in the same row or column of a table.
[0026] The header relationship (or header_of) indicates the relationship between the header cell and its corresponding series of data cells.
[0027] Label relationships (label_for) indicate the relationship between the axis labels of a chart and their corresponding bars or data points.
[0028] S23, Based on the target heterogeneous node set and the target heterogeneous edge set, generate the target heterogeneous document graph, which can be further understood as: the target heterogeneous document graph is a knowledge graph constructed from several target heterogeneous nodes and target heterogeneous edges between any two target heterogeneous nodes.
[0029] S3, input the target task and the target heterogeneous document graph into a preset large model to obtain the target question and answer text corresponding to the target task.
[0030] Specifically, step S3 also includes the following steps to obtain a preset large model:
[0031] S31, Obtain an initial sample document image set, which includes N0 initial sample document images, wherein the initial sample document images are scanned images of known public documents, and N0 is greater than 11 million, which can enrich the initial sample document image set.
[0032] S32, the initial sample document graph set is divided to generate a first sample document graph set and a second sample document graph set; wherein, the ratio between the number of first sample document graphs in the first sample document graph set and the number of second sample document graphs in the second sample document graph set is 19:1; further understood as: the first sample document graph set is used to train the model, and the second sample document graph set is used to validate the model.
[0033] S33, the initial large model is trained using the first sample document graph set to obtain an intermediate large model.
[0034] Furthermore, the structure of the initialized large model is a graph-biased multimodal Transformer structure.
[0035] Furthermore, step S33 also includes the following steps:
[0036] S331, input the heterogeneous graph corresponding to each first sample document graph into the initialized large model, and encode the multimodal features extracted from the heterogeneous graph corresponding to the first sample document graph through the graph bias attention mechanism to obtain the structured feature vector of the first sample document.
[0037] Preferably, the method for obtaining the heterogeneous graph corresponding to the first sample document graph is the same as the method for obtaining the target heterogeneous document graph corresponding to the target document graph, and will not be repeated here.
[0038] Preferably, the graph bias attention mechanism meets the following conditions:
[0039] .
[0040] Where B is the graph bias matrix, the input sequence (containing embedding vectors of elements such as text tokens and image patches) will undergo different linear transformations and be projected into the query matrix Q, the key matrix K and the value matrix V respectively, and dk is the dimension of the key vector.
[0041] Furthermore, B includes bias values Bij for several pairs of target heterogeneous nodes, where B ij It refers to the structural relationship strength between the i-th target heterogeneous node and the j-th other target heterogeneous node corresponding to the i-th target heterogeneous node, where the value of i ranges from 1 to N1+N2, and the value of j ranges from 1 to (N1+N2)-1; further, other target heterogeneous nodes refer to other target heterogeneous nodes in the target heterogeneous node set other than the i-th target heterogeneous node.
[0042] Furthermore, the B ij The following conditions must be met:
[0043] , where relation(v i v j This includes the shortest path length and edge type information between the i-th target heterogeneous node vi and the j-th other target heterogeneous nodes.
[0044] As described above, the explicit graph structure information of the document is natively and learnably integrated into the core attention calculation process of Transformer. This allows the model to naturally follow the inherent logic and spatial structure of the document when engaging in multimodal information interaction, thus fundamentally solving the problems of structural information loss and attention diversion caused by the "flattening" of documents in traditional methods.
[0045] S332, based on the structured feature vector corresponding to each first sample document, perform masked language modeling task, masked image modeling task, word block alignment task, and graph structure reconstruction task to obtain the joint optimization model parameters corresponding to the initialized large model.
[0046] S333, Adjust the parameters of the initialized large model according to the joint optimization model parameters corresponding to the initialized large model to obtain the intermediate large model.
[0047] S34, Obtain the initial instruction dataset, and perform fine-tuning on the intermediate large model according to the initial instruction dataset to obtain the preset large model.
[0048] Specifically, step S34 also includes the following steps:
[0049] S341, Obtain the initial instruction dataset, wherein the initial instruction data includes, but is not limited to: DocVQA and InfographicVQA datasets for general document visual question answering, ChartQA dataset for chart question answering, WikiTableQuestions dataset for table question answering, and FUNSD and CORD datasets for form and receipt information extraction; the above datasets are original data from multiple public benchmark datasets to ensure the model's general performance on multiple tasks.
[0050] S342, each initial instruction data in the initial instruction dataset is converted into a corresponding initial instruction triplet, wherein the initial instruction triplet includes an initial instruction, an initial document, and an initial answer; further understood as follows: the initial instruction is natural language text describing the task, such as "Calculate the total sales in 2023 based on the chart below" or "Extract the merchant name and date from the receipt"; the initial document is a token sequence representation of the target heterogeneous document graph corresponding to the instruction. This representation integrates the document's text content, visual block embeddings, and graph structure information; the initial answer is the standard answer text corresponding to the initial instruction.
[0051] S343, the initial instruction triplet is input into the intermediate large model for training to obtain the preset large model; further understood as: the intermediate large model is trained as an autoregressive generator, whose training objective is: given the sequence of "instructions" and "documents", to generate text that is completely consistent with the sequence of "answers" on a text feature-by-text basis, wherein the training is achieved by minimizing the cross-entropy loss between the predicted sequence and the real answer sequence.
[0052] Through this fine-tuning process, the model effectively transfers and adapts the general graph structure perception and multimodal fusion capabilities learned in the pre-training stage to the scenario of answering specific user questions, thereby obtaining a pre-set large model that can understand instructions, perform reasoning based on complex documents, and generate accurate text answers.
[0053] In summary, a knowledge question answering method based on a graph-structured multimodal document understanding large model includes: S1, obtaining a target task and a target document graph corresponding to the target task; S2, generating a target heterogeneous document graph corresponding to the target document graph based on the target document graph; S3, inputting the target task and the target heterogeneous document graph into a preset large model to obtain the target question answer text corresponding to the target task; achieving higher accuracy and robustness on multiple public benchmark tasks (such as document question answering, chart question answering, and table understanding); exhibiting excellent generalization ability for novel and complex document layouts; and achieving simpler and more efficient system deployment and application through an end-to-end unified architecture.
[0054] While specific embodiments of the invention have been described in detail by way of example, those skilled in the art should understand that the examples are for illustrative purposes only and not intended to limit the scope of the invention. It should also be understood that various modifications can be made to the embodiments without departing from the scope and spirit of the invention. The scope of the invention is defined by the appended claims.
Claims
1. A knowledge-based question-answering method for a large-scale multimodal document understanding model based on graph structure, characterized in that, The method includes the following steps: S1, Obtain the target task and the target document graph corresponding to the target task; S2, Generate a target heterogeneous document graph corresponding to the target document graph based on the target document graph; S3, input the target task and the target heterogeneous document graph into a preset large model to obtain the target question and answer text corresponding to the target task.
2. The knowledge question answering method for a large-scale multimodal document understanding model based on graph structure according to claim 1, characterized in that, Step S2 also includes the following steps: S21, perform node processing on the target document graph to obtain a target heterogeneous node set of the target document graph, wherein the target heterogeneous node set includes a first heterogeneous node set and a second heterogeneous node set; S22, perform edge processing on the target document graph to obtain the target heterogeneous edge set of the target document graph, wherein the target heterogeneous edge set includes N3 target heterogeneous edges, and the target heterogeneous edge is the edge between any two target heterogeneous nodes, wherein the target heterogeneous node is the first heterogeneous node or the second heterogeneous node. S23, Based on the target heterogeneous node set and the target heterogeneous edge set, generate the target heterogeneous document graph.
3. The knowledge question answering method for a large-scale multimodal document understanding model based on graph structure according to claim 2, characterized in that, The first heterogeneous node set includes N1 first heterogeneous nodes, wherein the first heterogeneous nodes are target text features extracted from the target document graph through a preset OCR model.
4. The knowledge question answering method for a large-scale multimodal document understanding model based on graph structure according to claim 3, characterized in that, The second heterogeneous node set includes N2 second heterogeneous nodes, wherein the second heterogeneous nodes are the non-textual features of the target document graph extracted by the preset target detection model.
5. The knowledge question answering method for a large-scale multimodal document understanding model based on graph structure according to claim 1, characterized in that, Step S3 also includes the following steps to obtain the preset large model: S31, Obtain an initial sample document image set, the initial sample document image set includes N0 initial sample document images, wherein the initial sample document images are scanned images of known public documents and N0 is greater than 11 million; S32, Divide the initial sample document graph set to generate the first sample document graph set and the second sample document graph set; S33, The initial large model is trained using the first sample document graph set to obtain an intermediate large model; S34, Obtain the initial instruction dataset, and perform fine-tuning on the intermediate large model according to the initial instruction dataset to obtain the preset large model.
6. The knowledge question answering method for a large-scale multimodal document understanding model based on graph structure according to claim 5, characterized in that, The ratio between the number of first sample document images in the first sample document image set and the number of second sample document images in the second sample document image set is 19:
1. S33, the initial large model is trained using the first sample document graph set to obtain an intermediate large model.
7. The knowledge question answering method for a large-scale multimodal document understanding model based on graph structure according to claim 5, characterized in that, The structure of the initialized large model is a graph-biased multimodal Transformer structure.