A visual document content extraction and analysis system and method

Abstract

The present invention discloses a visual document content extraction and analysis system and method, which includes a multi-task unified modeling module and a content extraction and information extraction decoding module. The multi-task unified modeling module includes a multimodal feature extraction submodule, a multimodal representation optimization submodule, and a task prompt generation submodule. The content extraction and information extraction decoding module includes the content extraction and information extraction decoding module and a unified decoding submodule. The method includes the following steps: S1, extracting multimodal features of a document image through the multi-task unified modeling module and generating a task prompt vector; S2, based on the multimodal features and the task prompt vector, using the content extraction and information extraction decoding module to synchronously complete document region parsing, semantic label assignment, and structured information output. The unified modeling of the present invention realizes the unified processing of multiple tasks, effectively solving the efficiency loss and consistency problems caused by task separation in traditional methods.

Description

Technical Field

[0001] The present invention relates to the interdisciplinary field of computer vision and natural language processing, and in particular to a system and method for multimodal content extraction and structured analysis of complex document images. Background Art

[0002] Most current automatic document reading and comprehension systems first extract text information using optical character recognition (OCR) technology, then perform semantic analysis of the document based on natural language processing methods. However, for visually rich documents (i.e., documents containing rich visual elements such as formulas, tables, and charts), existing document analysis and comprehension methods face two challenges: First, they struggle to recognize and understand complex non-text objects. Current methods, while mostly focused on visual text recognition, have significant shortcomings in understanding the semantics of non-text objects. These shortcomings include difficulty capturing the relative positional relationships and mathematical semantics of symbols in formulas, ineffectively identifying vertical and horizontal line structures, cross-cell merging logic, and the relationship between tables and contextual descriptions. Furthermore, they lack the ability to model the relative sizes of chart elements and data mapping relationships. Consequently, key semantic information about non-text objects cannot be fully extracted. Second, they suffer from task separation and model fragmentation. Existing technologies employ a phased modeling approach, first analyzing the document to extract sub-regions, and then further recognizing these sub-regions. For non-text objects, existing methods usually model and train them in stages, first extracting basic visual components and then combining the components for multimodal modeling and prediction, which leads to parameter redundancy, complex maintenance, and ineffective utilization of cross-modal correlation information.

[0003] Therefore, faced with large-scale document data, how to extract semantic information from visual content and establish a unified visual document object analysis and understanding mechanism to achieve rich visual document analysis and understanding is a key issue that needs to be solved urgently.

[0004] Therefore, the present invention is just based on above deficiency and produces. Summary of the Invention

[0005] The purpose of the present invention is to overcome the deficiencies of the prior art and provide a visual document content extraction and analysis system.

[0006] The present invention is achieved through the following technical solutions:

[0007] A visual document content extraction and analysis system is characterized by: including a multi-task unified modeling module and a content extraction and information extraction decoding module, the multi-task unified modeling module including a multimodal feature extraction submodule that can extract multimodal features from an input document image and perform pixel mask reconstruction prediction and context mining on the input document image through visual language mask prediction technology, a multimodal representation optimization submodule that uses contrastive learning to fuse vision, text, and position information to generate a unified multimodal representation, a task prompt generation submodule that generates task prompt vectors through prompt learning and unifies different tasks into a sequence generation and parsing framework, the content extraction and information extraction decoding module including a multimodal context fusion submodule that combines visual features, text semantics, and layout information to generate a unified semantic representation, and a unified decoding submodule that directly corresponds document regions to semantic labels through a shared decoder and uses a dynamic weight joint loss function to balance region detection and semantic allocation tasks.

[0008] The visual document content extraction and analysis system as described above is characterized in that the multimodal representation optimization submodule achieves adaptation to domain-specific tasks by inserting an adapter module.

[0009] The visual document content extraction and analysis system as described above is characterized in that: the visual language mask prediction technology includes pixel-level mask processing of text, tables, formulas, and illustration areas in document images and reconstructing the structure and content semantic information of the mask area through cross-modal contextual relationship mining.

[0010] The visual document content extraction and analysis system as described above is characterized in that the specific steps of the shared decoder to achieve end-to-end decoding are: generating the bounding box coordinates of the document area based on the unified semantic representation; assigning a semantic label to each area, the label including at least one of the title, text, table, formula, and illustration; and synchronously optimizing the area detection and semantic assignment tasks through the dynamic weight joint loss function.

[0011] The visual document content extraction and analysis system as described above is characterized in that: the dynamic weighted joint loss function includes region detection loss and semantic classification loss, and the weights are dynamically adjusted according to the training process to balance the region detection and semantic allocation tasks.

[0012] A method for extracting and analyzing visual document content, comprising the following steps:

[0013] S1. Extract multimodal features of document images through a multi-task unified modeling module and generate task prompt vectors;

[0014] S2. Based on the multimodal features and the task prompt vector, the content extraction and information extraction decoding modules are used to synchronously complete document region parsing, semantic tag assignment, and structured information output.

[0015] The above-mentioned method for extracting and analyzing visual document content is characterized in that step S1 includes:

[0016] S11. Extract multimodal features from the input document image and perform pixel mask reconstruction prediction and context mining on the input document image using visual language mask prediction technology;

[0017] S12, insert the adapter module to fuse visual, textual and position information and optimize the multimodal representation space;

[0018] S13. Generate task hint vectors through hint learning, unifying multimodal recognition tasks into a sequence generation and parsing framework.

[0019] The above-mentioned method for extracting and analyzing visual document content is characterized in that step S2 includes:

[0020] S21. Use the multimodal context modeling module to fuse visual features, text semantics, and layout information to generate a unified semantic representation;

[0021] S22. Use a shared decoder to map document regions to semantic labels, and use a dynamic weighted joint loss function to balance region detection and semantic assignment.

[0022] The visual document content extraction and analysis method as described above is characterized in that it also includes step S3: collecting rich visual documents through diversified data sources, generating a multimodal document test set and annotating it through a grid candidate layout algorithm; S4: using the multimodal document test set to verify the multi-tasking processing capability and robustness of the model.

[0023] The visual document content extraction and analysis method as described above is characterized in that the grid candidate layout algorithm includes candidate sampling, grid construction, best match search and iterative layout filling.

[0024] Compared with the prior art, the present invention has the following advantages:

[0025] 1. First, the unified modeling of the present invention realizes the unified processing of multiple tasks, aligning tasks such as table recognition, formula recognition, chart recognition, text recognition, and illustration recognition under the same multimodal representation framework. By sharing and reusing model parameters, the collaborative efficiency between different tasks and the generalization performance of the model are significantly improved. Secondly, through the synchronous decoding framework of content extraction and information extraction, an efficient mapping from document content extraction to structured semantic information allocation is successfully achieved, enabling the present invention to simultaneously complete multimodal task processing and in-depth analysis of semantic information in complex document scenarios, effectively solving the efficiency loss and consistency problems caused by task separation in traditional methods, and providing a unified, efficient, and robust solution for the analysis of complex documents.

[0026] 2. The present invention also uses a high-quality document test set covering multimodal content such as text, tables, formulas, illustrations, emoticons, etc. to verify the multi-tasking processing capability and robustness of the model, ensuring that the visual document content extraction and analysis system can not only perform well on the training data, but also adapt to actual complex scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 is a schematic diagram of the visual document content extraction and analysis system of the present invention;

[0028] Figure 2 Schematic diagram of the modeling method of the multi-task unified modeling module of the present invention;

[0029] Figure 3 Schematic diagram of the end-to-end content extraction and information extraction decoding method of the present invention;

[0030] Figure 4 Schematic diagram of the grid candidate best matching algorithm of the present invention. DETAILED DESCRIPTION

[0031] The present invention will be further described below in conjunction with the accompanying drawings:

[0032] like Figure 1 As shown, a visual document content extraction and analysis system includes a multi-task unified modeling module and a content extraction and information extraction decoding module.

[0033] In order to achieve efficient sharing and parsing of multiple tasks, the multi-task unified modeling module includes a multimodal feature extraction sub-module, which can extract multimodal features from the input document image, such as text paragraphs, tables, formulas, illustrations, emoticons, etc., and perform pixel mask reconstruction prediction on the input document image through visual language mask prediction technology, as well as mining visual context information through context mining to enhance the robustness of feature extraction and semantic representation capabilities; a multimodal representation optimization sub-module, which uses contrastive learning to fuse vision and text as well as position information to generate a unified multimodal representation and optimize the multimodal representation space; a task prompt generation sub-module, which generates task prompt vectors through prompt learning and unifies different tasks including table recognition, formula recognition, chart recognition, text recognition and illustration recognition into a sequence generation and parsing framework, solving the complexity problem brought about by task separation in traditional methods.

[0034] In order to achieve end-to-end content extraction and information extraction decoding, the content extraction and information extraction decoding module includes a multimodal context fusion sub-module, which uses the multimodal context modeling module to combine visual features, text semantics and layout information to generate a unified semantic representation. Based on the extracted features, it completes the accurate parsing of the document area content, effectively solving the parsing problems caused by complex layout and cross-modal associations; the unified decoding sub-module directly corresponds the document area to semantic labels (such as "title", "text", "table", etc.) through a shared decoder and uses a dynamic weight joint loss function to balance the area detection and semantic allocation tasks, thereby reducing error accumulation and improving parsing efficiency.

[0035] To meet the needs of different tasks, the multimodal representation optimization submodule achieves domain-specific adaptation by inserting adapter modules. Small adapter modules are inserted between each Transformer layer in the pre-trained model to deeply integrate visual, textual, and positional information, optimizing the multimodal representation space. In practical applications, this approach effectively preserves general knowledge while adapting to domain-specific tasks, significantly improving robustness and adaptability.

[0036] Visual language mask prediction technology involves pixel-level masking of text, tables, formulas, and illustrations in document images and reconstructing the structure and content semantic information of the masked areas through cross-modal contextual relationship mining.

[0037] The task prompt generation submodule uses a learnable prompt vector to unify the parsing process of different tasks into a sequence generation method based on the Transformer architecture.

[0038] The specific steps of the shared decoder to achieve end-to-end decoding are: generating the bounding box coordinates of the document area based on the unified semantic representation; assigning a semantic label to each area, wherein the label includes at least one of the title, text, table, formula, and illustration, and using a Softmax classifier to output the category probability; and synchronously optimizing the region detection and semantic assignment tasks through a dynamic weighted joint loss function.

[0039] The dynamic weighted joint loss function includes region detection loss and semantic classification loss. The weights are dynamically adjusted according to the training process to balance the region detection and semantic assignment tasks. The weights are automatically adjusted according to the task uncertainty.

[0040] The present invention also discloses a method for extracting and analyzing visual document content, comprising the following steps:

[0041] S1. Extract multimodal features of document images through a multi-task unified modeling module and generate task prompt vectors;

[0042] S2. Based on the multimodal features and the task prompt vector, the content extraction and information extraction decoding modules are used to synchronously complete document region parsing, semantic tag assignment, and structured information output.

[0043] Step S1 includes:

[0044] S11. Extract multimodal features from the input document image and perform pixel mask reconstruction prediction and context mining on the input document image using visual language mask prediction technology;

[0045] S12, insert the adapter module to fuse visual, textual and position information and optimize the multimodal representation space;

[0046] S13. Generate task hint vectors through hint learning, unifying multimodal recognition tasks into a sequence generation and parsing framework.

[0047] Step S2 includes:

[0048] S21. Use the multimodal context modeling module to fuse visual features, text semantics, and layout information to generate a unified semantic representation;

[0049] S22. Use a shared decoder to map document regions to semantic labels, and use a dynamic weighted joint loss function to balance region detection and semantic assignment.

[0050] The present method for extracting and analyzing visual document content also includes step S3: collecting rich visual documents through diversified data sources, generating and annotating a multimodal document test set through the grid candidate layout algorithm; S4: using the multimodal document test set to verify the multitasking processing capability and robustness of the model. Step S3 is performed before or simultaneously with step S1, and step S4 is performed after step S2. Step S3 is the data preparation stage, which aims to use the grid candidate layout algorithm such as Figure 4 As shown, a high-quality, multimodal document test set is generated, and combined with precise annotation to ensure the diversity and applicability of the training data. S1 and S2 are the model training stages, which aim to input document images into the multi-task unified modeling module to complete multimodal feature extraction and task prompt learning. On this basis, the decoding module is used to simultaneously optimize content extraction and semantic label assignment, and a dynamically adjusted joint loss function is used to ensure multi-task collaboration and performance improvement. Step S4 is the model verification stage, which aims to use the test set to verify the multi-task processing capability and robustness of the model, and further verify the model performance through actual application scenarios such as social media content analysis.

[0051] Specifically, the grid candidate layout algorithm includes candidate sampling, grid construction, best match search, and iterative layout filling. Candidate sampling involves selecting a diverse range of elements, such as text, tables, formulas, and illustrations, from a document element library. Grid construction involves constructing an adaptive grid structure based on the document type and defining layout constraints. Best match search searches for the optimal grid position for each element based on semantic relevance and layout rules. Iterative layout filling uses an optimization algorithm to iteratively adjust element positions to maximize the layout score and generate a high-quality test set.

[0052] In general, the unified modeling method and synchronous decoding framework proposed in the present invention can efficiently handle multimodal tasks in complex documents and achieve simultaneous optimization of content extraction and semantic information extraction. It is mainly composed of a unified modeling module and a content extraction and information extraction decoding module. The trained model can be directly applied to different document parsing tasks without the need for special task customization. The model has strong generalization and transferability, and is applicable to various document types, such as scanned documents, social media screenshots, scientific papers, etc., without the need to design a dedicated output header for each task. The trained model can not only share and reuse parameters in different task scenarios, but also the prediction accuracy is positively correlated with the model size. As the model size increases, the performance is significantly improved, which provides broad space for further optimization and development of the model. Although training requires large datasets, these datasets are easy to obtain, and through the automated pairing of multimodal data sources, a large-scale, high-quality document test set can be quickly constructed. In addition, the unified modeling method of the present invention solves the complexity problem caused by multi-task separation. Through task sharing and optimization, it greatly improves the collaborative efficiency between tasks and can handle non-text objects in complex documents, such as tables, formulas, illustrations, etc. This method demonstrates the advantages of the present invention in multimodal tasks and provides strong technical support for future applications in a wider range of fields, such as document analysis and sentiment analysis. In summary, the present invention not only improves the efficiency of multitasking but also enhances the generalization ability and applicability of the model, providing an innovative solution for complex document parsing and multimodal tasks.

Claims

1. A method for extracting and analyzing visual document content, characterized in that: The following steps are involved: S1. Extract multimodal features of document images through a multi-task unified modeling module and generate task prompt vectors; S2. Based on the multimodal features and the task prompt vector, using the content extraction and information extraction decoding modules to simultaneously complete document region parsing, semantic label assignment, and structured information output; The step S1 includes S11, extracting multimodal features from the input document image and performing pixel mask reconstruction prediction and context mining on the input document image through visual language mask prediction technology; S12, integrating visual, textual and location information to optimize the multimodal representation space; S13, generate task hint vectors through hint learning, and unify multimodal recognition tasks into a sequence generation and parsing framework; The step S2 includes S21, fusing visual features, text semantics and layout information to generate a unified semantic representation; S22. Use a shared decoder to map document regions to semantic labels, and use a dynamic weighted joint loss function to balance region detection and semantic assignment.

2. The method for extracting and analyzing visual document content according to claim 1, wherein: It also includes step S3: collecting rich visual documents through diverse data sources, generating and annotating a multimodal document test set through a grid candidate layout algorithm; S4: using the multimodal document test set to verify the multi-task processing capability and robustness of the model.

3. The method for extracting and analyzing visual document content according to claim 2, wherein: The grid candidate layout algorithm includes candidate sampling, grid construction, best match search and iterative layout filling.

4. A visual document content extraction and analysis system, the system being used to implement the visual document content extraction and analysis method according to any one of claims 1 to 3, characterized in that: It includes a multi-task unified modeling module and a content extraction and information extraction decoding module. The multi-task unified modeling module includes a multimodal feature extraction submodule that can extract multimodal features from input document images and perform pixel mask reconstruction prediction and context mining on the input document images through visual language mask prediction technology, a multimodal representation optimization submodule that uses contrast learning to fuse vision, text and position information to generate a unified multimodal representation, and a task prompt generation submodule that generates task prompt vectors through prompt learning and unifies different tasks into a sequence generation and parsing framework. The content extraction and information extraction decoding module includes a multimodal context fusion submodule that combines visual features, text semantics and layout information to generate a unified semantic representation, and a unified decoding submodule that directly corresponds document areas to semantic labels through a shared decoder and uses a dynamic weight joint loss function to balance area detection and semantic allocation tasks.

5. The visual document content extraction and analysis system according to claim 4, characterized in that: The multimodal representation optimization submodule realizes the adaptation of domain-specific tasks by inserting an adapter module.

6. The visual document content extraction and analysis system according to claim 4, characterized in that: The visual language mask prediction technology includes pixel-level mask processing of text, tables, formulas, and illustration areas in document images and reconstructing the structure and content semantic information of the masked areas through cross-modal contextual relationship mining.

7. The visual document content extraction and analysis system according to claim 4, characterized in that: The specific steps of the shared decoder to achieve end-to-end decoding are: generating the bounding box coordinates of the document area based on the unified semantic representation; assigning a semantic label to each area, wherein the label includes at least one of the title, text, table, formula, and illustration; and synchronously optimizing the region detection and semantic assignment tasks through the dynamic weight joint loss function.

8. The visual document content extraction and analysis system according to claim 4, characterized in that: The dynamic weighted joint loss function includes region detection loss and semantic classification loss, and the weights are dynamically adjusted according to the training process to balance the region detection and semantic assignment tasks.

Images