End-to-end document classification and information extraction system, method and readable storage medium based on multi-modal large model

Abstract

The application discloses an end-to-end document classification and information extraction system and method based on a multi-modal large model and a readable storage medium, and relates to the field of artificial intelligence and deep learning.The system comprises the following steps: receiving an input document image, performing image enhancement to obtain a pretreated image; performing document classification by using a first multi-modal large model that is fine-tuned, determining a document category and taking the document category as a preset category; inputting an information extraction image, performing document classification by using the first multi-modal large model that is fine-tuned, selecting an information extraction strategy, responding to an information extraction request instruction based on a second multi-modal large model that is fine-tuned, and forming and outputting a structured extraction result.The application is designed for multiple types of document scenarios, can maintain a stable processing flow under the condition that the number of document types is relatively large and continuously changes, and supports automatic processing of untrained document types.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and deep learning, and in particular to an end-to-end document classification and information extraction system, method and readable storage medium based on a multimodal large model. Background Technology

[0002] With the advancement of informatization, the automatic recognition and structured processing of document images have been widely applied in fields such as finance, government affairs, taxation, healthcare, and enterprise management. In actual business operations, a large number of certificates, invoices, contracts, and forms exist in the form of scanned or photographed copies. How to automatically extract structured information from these documents has become a crucial technical issue for improving business processing efficiency. Existing document processing systems typically include two stages: document classification and information extraction. Document classification is used to determine the document type, while information extraction is used to extract field or table information from documents of known types. However, in practical applications, the number of document types is numerous and constantly changing, and documents from different sources vary significantly in format and content, posing a significant challenge to document processing systems. In terms of document classification, existing technologies mostly use rules, templates, or supervised learning models to directly output category labels. These methods usually rely on a fixed set of training categories. When document types not covered in the training set appear, the classification accuracy drops significantly, and adding new document types often requires retraining the model, resulting in high system expansion costs. In terms of information extraction, existing technologies mostly configure field rules or train dedicated models for specific document types, resulting in poor versatility. When the document type is uncertain or changes, the extracted configuration is difficult to adapt, affecting system stability.

[0003] In recent years, although multimodal large models have been introduced into the field of document understanding, existing solutions mostly use a single model to directly extract information, lacking a systematic mechanism for handling document type uncertainty and open scenarios. When the input document type exceeds the model's training range, the extraction results are unstable, and there is a lack of effective fallback mechanisms. Furthermore, in existing document processing systems, document classification and information extraction are usually independent, with classification results used only as static labels. The extraction strategy cannot be dynamically adjusted based on document features, leading to potential interruptions in end-to-end processing when classification is inaccurate or the document is unknown.

[0004] Therefore, there is an urgent need for an end-to-end document processing technology that can adapt to the continuous expansion of document types and still stably output structured information in scenarios without training documents. Summary of the Invention

[0005] This invention provides an end-to-end document classification and information extraction system, method, and readable storage medium based on a multimodal large model. Designed for multi-type document scenarios, this invention can maintain a stable processing flow even when the number of document types is large and constantly changing, and supports automatic processing of untrained document types.

[0006] According to a first aspect of the technical solution of the present invention, a document classification and information extraction method based on a multimodal model is provided, comprising: S1, Receive an input document image, perform image enhancement processing on the input document image, and obtain a preprocessed image; S2: For the preprocessed image, a fine-tuned first multimodal large model is used for document classification. The document image is represented as a continuous structural semantic vector. Template vectors are retrieved from the vector database for similarity matching to determine the document category and use it as a preset category. S3: Input the image of information to be extracted, use the fine-tuned first multimodal large model to classify the image of information to be extracted into documents, select an information extraction strategy according to whether the document category belongs to the preset category, respond to the information extraction request instruction based on the fine-tuned second multimodal large model, and form and output the structured extraction result.

[0007] Furthermore, in step S1, if the input is a PDF document, the format is converted to split the PDF document into single-page images, and then image enhancement processing is performed.

[0008] Furthermore, in S1, the image enhancement processing includes rotation and scaling, affine transformation and geometric deformation, lighting simulation and color change, and simulation of wet and stain effects.

[0009] Furthermore, S1 specifically includes: The input document image is rotated and scaled to obtain a first processed image; Apply affine transformation and geometric deformation to the first processed image to obtain the second processed image; The second processed image is subjected to lighting simulation and color change processing to obtain the third processed image; Apply a wetness effect and a stain effect to the third processed image to obtain the preprocessed image.

[0010] Furthermore, the first / second multimodal models include Qwen2.5-vl, Dots.ocr, Hunyuanocr, Paddleocr-vl, etc.

[0011] Further, in S2, the fine-tuned first multimodal model includes: The first multimodal model is trained using contrastive learning, with samples of the same class as positive samples and samples of different classes as negative samples, and the similarity of normalized feature vectors is calculated. A normalized probability distribution is constructed based on the similarity, and the log loss is calculated using the similarity terms corresponding to samples of the same category as positive sample terms. When there are multiple positive samples, the loss results corresponding to multiple positive samples are averaged. The loss values ​​of all samples are summarized to obtain the loss function used for updating model parameters.

[0012] Further, in step S2, retrieving template vectors from the vector database for similarity matching to determine the document category includes: Sample images are provided for each document category, and template vectors are generated by encoding the first multimodal model that has been fine-tuned, forming a vector database; When performing similarity matching, the template vector that is most similar to the document classification result in the database is retrieved to determine the corresponding document category.

[0013] Furthermore, in step S2, when a document type does not appear in the vector database, a sample image of the same type is provided and encoded by the fine-tuned first multimodal model to generate a template vector, which is then added to the vector database to achieve classification support for the new document type.

[0014] Furthermore, in S3, the information extraction request instruction includes a composite extraction request instruction and a single extraction request instruction, and the composite extraction request instruction can be split into multiple sub-requests with the same structure as the single extraction request.

[0015] Furthermore, in step S3, when the information extraction request instruction is a composite extraction request instruction: The composite extraction request instruction is parsed at the task level and split into multiple sub-requests with the same structure as a single extraction request. Each sub-request is input into the finely tuned second multimodal large model for parallel inference processing to obtain the extraction result corresponding to each sub-request. The extraction results from each sub-request are summarized to form the final structured extraction result.

[0016] Furthermore, the structured extraction results include: Summarize the extraction results of each sub-request, and concatenate key-value pair information, header data, and table content; The coordinate information of each element is retained to form complete structured data.

[0017] Furthermore, in S3, the fine-tuned second multimodal large model includes: The pre-training of the second multimodal large model includes document parsing tasks, supporting the output of entity positions, logical table alignment information, and complete physical table structure, generating parsing results that cover all extraction targets.

[0018] Furthermore, in S3, the objects extracted include: entities, logical tables, and physical tables; Among them, entities are key-value pairs in the document; logical tables are well-structured, row-and-column aligned tables in the document, from which information is extracted; and physical tables are any complete tables in the document, from which information is extracted based on the table name.

[0019] Furthermore, in step S3, the information extraction strategy includes: When the document category of the image to be extracted belongs to the preset category, a targeted information extraction mode is performed using preset keywords corresponding to the preset category; When the document category of the image to be extracted does not belong to any preset category, the intelligent extraction mode is adopted to extract all information from the document content based on the general extraction instructions.

[0020] Furthermore, the intelligent extraction mode specifically includes: The general extraction command and the image of information to be extracted are input into the fine-tuned second multimodal large model; The finely tuned second multimodal large model is used to perform traversal analysis on the image of information to be extracted, and to identify and output all information units in the document that match the type of the object to be extracted. If no information unit matching the extraction target type is detected, a feedback message indicating that the extraction target does not exist will be output. The general extraction instruction is used to instruct the analysis of the entire document content, without limiting the specific fields to be extracted.

[0021] According to a second aspect of the present invention, an end-to-end document classification and information extraction system based on a multimodal large model is provided, the system comprising: a processor and a memory for storing executable instructions; wherein the processor is configured to execute the executable instructions to perform the end-to-end document classification and information extraction method based on a multimodal large model as described in any of the preceding aspects.

[0022] According to a third aspect of the present invention, a computer-readable storage medium is provided, wherein a computer program is stored thereon, and when the computer program is executed by a processor, it implements the end-to-end document classification and information extraction method based on a multimodal large model as described in any of the preceding aspects.

[0023] The beneficial effects of this invention are: 1. This invention realizes an end-to-end document processing system, in which users only need to input the document to automatically obtain structured information output without additional input, thereby improving the automation level and ease of use of the system.

[0024] 2. This invention adopts a document classification method based on structural semantic vectors, which avoids dependence on fixed training categories and enables the system to expand document types during the runtime phase with a small number of template vectors, significantly improving the system's adaptability to multiple types and new document scenarios.

[0025] 3. Based on the matching of document vectors and template vectors, this invention adaptively switches information extraction strategies, and can still correctly classify new document types that have not been trained, stably outputting structured information, thus improving the robustness of the end-to-end processing flow.

[0026] 4. This invention adopts a unified information extraction model, which supports multi-level extraction of entity information, logical table information and physical table information, and can directly output location information, thereby enhancing the practicality of the extraction results in display, verification and subsequent processing.

[0027] 5. This invention supports splitting user extraction requests, which can split composite requests containing multiple extraction targets into multiple sub-requests for concurrent execution, improving the system's ability to support complex extraction needs while ensuring the efficiency and scalability of the extraction process. Attached Figure Description

[0028] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the structures shown in these drawings without creative effort.

[0029] Figure 1 The flowchart of an end-to-end document classification and information extraction system based on a multimodal large model is shown.

[0030] Figure 2 A sample diagram is shown.

[0031] Figure 3 The diagram shows an example of prompt splitting.

[0032] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0033] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0034] The terms "first," "second," etc., used in this disclosure are for distinguishing similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented, for example, in orders other than those illustrated or described herein.

[0035] Furthermore, the terms “comprising” and “having”, and any variations thereof, are intended to cover non-exclusive inclusion, such that a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such process, method, product, or apparatus.

[0036] Multiple, including two or more.

[0037] And / or, it should be understood that, for the purposes of this disclosure, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone.

[0038] Example This invention relates to an end-to-end processing method based on a multimodal large model that deeply integrates document image classification and structured information extraction. The system automatically extracts structured information from the document content after the user inputs it. In existing document processing systems, multi-type document recognition and information extraction rely on fixed training models, which cannot adapt to the continuous expansion of document types. When encountering untrained document types, they often fail to correctly classify and extract structured information.

[0039] like Figure 1 As shown, this invention mainly includes a document preprocessing module, a document classification module, and an information extraction module. Through structural semantic vector representation, template vector matching, and dynamic selection of extraction strategies, it achieves high-precision classification and flexible structured information extraction for multiple document types. Simultaneously, it supports rapid access and intelligent extraction of new document types, ensuring end-to-end robustness and scalability of the system.

[0040] 1. Document Preprocessing Module The main function of the document preprocessing module is to perform type conversion and image feature processing on the input document, ensuring that subsequent steps can be performed on the high-quality image. This module consists of the following two parts: 1) Document type conversion To ensure consistent document formatting, this section converts the input PDF file into an image file. For multi-page PDFs, they are split into multiple single-page documents according to page numbers, and each page is converted into its corresponding image file. Documents in other image formats, such as JPEG or PNG, proceed directly to the next image processing step.

[0041] 2) Image Feature Processing To improve the model's adaptability to different shooting conditions and document states, this invention performs various forms of enhancement processing on document images during the model training phase to increase the diversity and complexity of the training data. The image processing includes, but is not limited to, the following methods: rotating and scaling the document image to simulate changes in shooting angle and resolution; applying affine transformations and geometric deformations to the document image to simulate deformations or non-ideal shooting postures that occur during actual use; simulating different ambient lighting conditions and differences in imaging equipment through lighting simulation and color change processing; and simulating creases, stains, or partial occlusions that may occur in documents during actual storage and use through moisture and stain effects.

[0042] By combining the above-mentioned image enhancement methods, the model can learn feature representations that are more robust to document content structure and semantic information, thereby improving the stability and accuracy of the system in document classification and information extraction tasks in complex real-world scenarios.

[0043] 2. Document Classification Module A vectorized document classification mechanism based on a multimodal large model is adopted to transform the document classification results from discrete category labels into continuous structural semantic vector representations. The definition and expansion of document categories are completed through a small number of template vectors, enabling the system to support the access of new document types during the runtime phase without the need to retrain the classification model.

[0044] 1) Fine-tuning the multimodal large model. The multimodal large model mentioned in this module refers to a pre-trained multimodal feature encoding model that jointly models input data combining text and images and outputs vector representations in a unified semantic space. Pre-training tasks include multimodal large models for document parsing tasks, such as Qwen2.5-vl, Dots.ocr, Hunyuanocr, and Paddleocr-vl. To obtain high-quality image vector representations and distinguish different types of documents, in one embodiment of this invention, the model is trained based on a contrastive learning approach. Specifically, in each training batch, based on the category label corresponding to the sample, other samples of the same category as the current sample are identified as positive samples, and samples of different categories are identified as negative samples. After normalizing the feature vectors obtained from the large model, the similarity between the current sample and other samples in the same batch is calculated. In this process, a temperature coefficient is introduced to scale the similarity, thereby adjusting the sharpness of the distinction between positive and negative samples and controlling the intensity of intra-class aggregation and inter-class separation. A normalized probability distribution is constructed based on the similarity, and the logarithmic loss is calculated using the similarity terms corresponding to samples of the same category as positive sample terms. When there are multiple positive samples, the loss results corresponding to multiple positive samples are averaged. The loss values ​​of all samples in the training batch are summarized to obtain the loss function used for updating model parameters. Through the above method, samples of the same category are clustered in the feature space, and samples of different categories are separated in the feature space, improving the model's discriminative ability. The contrastive loss function can be, but is not limited to, expressed in the following form: in, This represents the sample index in the training batch. Representation and Sample A set of positive samples of the same category Indicates sample Compared with positive samples Similarity between them Indicates sample Similarity with other samples in the training batch This is a temperature coefficient used to adjust the smoothness of the similarity distribution. By normalizing the positive sample similarity term and calculating the log loss, samples of the same class are more clustered in the feature space, while samples of different classes form larger feature gaps.

[0045] 2) Template Vector Generation and Storage. Before inference, the user selects 1 to 3 sample document images as templates for each document category. The system uses a fine-tuned multimodal large model to encode the features of the template documents, generating high-dimensional template vectors. These template vectors are stored in a vector database as vector indexes for that category, used for subsequent similarity matching.

[0046] 3) Document Vector Generation and Matching. During inference, the user uploads the document to be processed, and the system calls a fine-tuned multimodal large model to generate the document's structural semantic vector. Similarity is then searched in the vector database to match the document category corresponding to the most similar template vector. When the document type is entirely new and not found in the system's templates, the user can provide a small number of template images for rapid expansion. The system adds these template vectors to the vector database to support the classification of new document types. Through the vector matching mechanism, the system does not rely on full training of predefined categories, supports the expansion of new document types, and the classification results can be directly used for information extraction control, ensuring the continuity and robustness of the end-to-end process.

[0047] 3. Information Extraction Module It supports splitting user extraction requests, breaking down composite requests containing multiple extraction targets into multiple sub-requests, and executing them concurrently using extraction instructions with consistent structure, thereby achieving flexible combination and unified output of multi-target information extraction.

[0048] 1) Fine-tuning the Multimodal Large Model. The multimodal large model mentioned in this module refers to a pre-trained multimodal large model that includes document parsing tasks. The system's information extraction module uses this fine-tuned multimodal large model to extract different types of information from the document and can output the coordinate information of each element. Extractable objects include entities, logical tables, and physical tables. Entities are key-value pairs in the document; logical tables are well-structured, row-and-column aligned tables in the document, and information is extracted from logical tables based on their headers; physical tables are any complete table in the document, including merged cells and nested tables, and information is extracted from physical tables based on their names. For example... Figure 2 As shown in the sample, the content in the red boxes is the entity, the content in the blue boxes is the logical table header, and the content in the green boxes is the physical table.

[0049] 2) Prompt splitting and parallel processing of sub-requests. In one embodiment of the present invention, the system supports users to submit composite extraction requests that include multiple extraction tasks.

[0050] During the model training phase, multiple extraction instructions are pre-constructed and used to train the model, including composite extraction instructions and single extraction task instructions. The composite extraction instructions and single extraction task instructions are consistent in terms of task description structure and output constraint method, and differ only in the number of extraction tasks they contain, so that the model can learn to complete different numbers of extraction tasks under a unified instruction structure.

[0051] During the inference phase, when a composite extraction request is received, the system performs task-level parsing on the composite extraction request based on the unified task structure, and splits the composite extraction request into several sub-requests consistent with the instruction structure of a single extraction task. The splitting process can be as follows: Figure 3 As illustrated in the case study, because the sub-requests are consistent with the instructions for a single extraction task in terms of instruction structure and model input format, the system can perform parallel inference processing on each sub-request, thereby shortening the overall inference time. The extraction results corresponding to each sub-request are summarized and concatenated within the system to form the final structured extraction result output.

[0052] 3) Intelligent extraction capability. In one embodiment of the present invention, the intelligent extraction is not based on preset fields, but is accomplished collaboratively by a general extraction instruction and a large model trained in a specific manner. Specifically, a general extraction instruction is input to the large model to instruct the extraction task. The general extraction instruction does not limit specific extraction fields, but is used to instruct the model to perform a traversal analysis of the entire content of the input sample and autonomously identify all information units in the sample that conform to the predetermined extraction type.

[0053] During the model training phase, a training message set containing various general extraction instructions is constructed. These general extraction instructions include, but are not limited to, "extract all entity information from the sample" and "extract all table content from the sample." For each general extraction instruction, a corresponding complete extraction result is provided as a supervision signal, enabling the model to learn to perform a global analysis of the input sample and output extraction results covering all target information without specifying a specific extraction target. For samples where no corresponding extraction target exists, negative response examples are further constructed in the training message, enabling the model to output preset feedback information indicating the absence of the target when it detects that the extraction target does not exist in the input sample.

[0054] During the inference phase, upon receiving a general extraction instruction, the model, based on its traversal analysis capabilities developed during training, performs overall feature parsing on the input sample and returns all information units in the sample that meet the extraction conditions according to predetermined output constraint rules. If no information unit that meets the conditions is detected, a response indicating that the extraction target does not exist is returned. This method achieves intelligent extraction functionality without the need for predefined fields.

[0055] The main innovative points of this invention are as follows: 1. Classification Vector-Driven Extraction Strategy: The structural semantic vector generated by the document classification module is used to control the strategy selection of the information extraction module: First, the document image is classified based on the document classification model, and targeted information extraction is performed using preset keywords for known types; when the document type is detected to be not in the preset category set, it automatically switches to the intelligent extraction mode to extract all information from the document content.

[0056] 2. Template scalability: Users can provide a small number of templates for new document categories at any time without retraining the model, enabling rapid category expansion.

[0057] 3. Request splitting and parallel extraction: The system can split complex requests into multiple sub-requests for parallel processing, improving processing efficiency and stability.

[0058] In summary, this invention discloses an information extraction and tracing method, system, and readable storage medium based on the collaboration of an OCR model and a large language model (LLM). It combines a small OCR model and a large language model (LLM) to extract textual information from images and trace back to the original image coordinates. This method enables structured extraction of textual information from images and precisely traces the extracted results to their location within the original image, achieving traceability and verifiability of the information.

[0059] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0060] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0061] Through the above description of the embodiments, those skilled in the art can clearly understand that the above implementation methods can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0062] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other modifications under the guidance of the present invention without departing from the spirit and scope of the claims. All of these modifications are within the protection scope of the present invention.

Claims

1. A document classification and information extraction method based on a multimodal model, characterized in that, include: S1, Receive an input document image, perform image enhancement processing on the input document image, and obtain a preprocessed image; S2: For the preprocessed image, a fine-tuned first multimodal large model is used for document classification. The document image is represented as a continuous structural semantic vector. Template vectors are retrieved from the vector database for similarity matching to determine the document category and use it as a preset category. S3: Input the image of information to be extracted, use the fine-tuned first multimodal large model to classify the image of information to be extracted into documents, select an information extraction strategy according to whether the document category belongs to the preset category, respond to the information extraction request instruction based on the fine-tuned second multimodal large model, and form and output the structured extraction result.

2. The document classification and information extraction method based on a multimodal model according to claim 1, characterized in that, In step S1, if the input is a PDF document, the format is converted to split the PDF document into single-page images, and then image enhancement processing is performed.

3. The document classification and information extraction method based on a multimodal model according to claim 1, characterized in that, In S1, the image enhancement processing includes rotation and scaling, affine transformation and geometric deformation, lighting simulation and color change, and simulation of wet and stain effects.

4. The document classification and information extraction method based on a multimodal model according to claim 1, characterized in that, In S2, the fine-tuned first multimodal model includes: The first multimodal model is trained using contrastive learning, with samples of the same class as positive samples and samples of different classes as negative samples, and the similarity of normalized feature vectors is calculated. A normalized probability distribution is constructed based on the similarity, and the log loss is calculated using the similarity terms corresponding to samples of the same category as positive sample terms. When there are multiple positive samples, the loss results corresponding to multiple positive samples are averaged. The loss values ​​of all samples are summarized to obtain the loss function used for updating model parameters.

5. The document classification and information extraction method based on a multimodal model according to claim 1, characterized in that, In step S2, retrieving template vectors from the vector database for similarity matching to determine the document category includes: Sample images are provided for each document category, and template vectors are generated by encoding the first multimodal model that has been fine-tuned, forming a vector database; When performing similarity matching, the template vector that is most similar to the document classification result in the database is retrieved to determine the corresponding document category.

6. The document classification and information extraction method based on a multimodal model according to claim 5, characterized in that, In step S2, when a document type does not appear in the vector database, a sample image of the same type is provided and encoded by the fine-tuned first multimodal model to generate a template vector, which is then added to the vector database to achieve classification support for the new document type.

7. The document classification and information extraction method based on a multimodal model according to claim 1, characterized in that, In step S3, the information extraction request instruction includes a composite extraction request instruction and a single extraction request instruction. The composite extraction request instruction can be divided into multiple sub-requests with the same structure as the single extraction request instruction.

8. The document classification and information extraction method based on a multimodal model according to claim 7, characterized in that, In step S3, when the information extraction request instruction is a composite extraction request instruction: The composite extraction request instruction is parsed at the task level and split into multiple sub-requests with the same structure as a single extraction request. Each sub-request is input into the finely tuned second multimodal large model for parallel inference processing to obtain the extraction result corresponding to each sub-request. The extraction results from each sub-request are summarized to form the final structured extraction result.

9. The document classification and information extraction method based on a multimodal model according to claim 8, characterized in that, The structured extraction results include: Summarize the extraction results of each sub-request, and concatenate key-value pair information, header data, and table content; The coordinate information of each element is retained to form complete structured data.

10. The document classification and information extraction method based on a multimodal model according to claim 1, characterized in that, In S3, the fine-tuned second multimodal large model includes: The pre-training of the second multimodal large model includes document parsing tasks, supporting the output of entity positions, logical table alignment information, and complete physical table structure, generating parsing results that cover all extraction targets.

11. The document classification and information extraction method based on a multimodal model according to claim 1, characterized in that, In S3, the objects extracted include: entities, logical tables, and physical tables; Among them, entities are key-value pairs in the document; logical tables are well-structured, row-and-column aligned tables in the document, from which information is extracted; and physical tables are any complete tables in the document, from which information is extracted based on the table name.

12. The document classification and information extraction method based on a multimodal model according to claim 1, characterized in that, In step S3, the information extraction strategy includes: When the document category of the image to be extracted belongs to the preset category, a targeted information extraction mode is performed using preset keywords corresponding to the preset category; When the document category of the image to be extracted does not belong to any preset category, the intelligent extraction mode is adopted to extract all information from the document content based on the general extraction instructions.

13. The document classification and information extraction method based on a multimodal model according to claim 12, characterized in that, The intelligent extraction mode is specifically as follows: The general extraction command and the image of information to be extracted are input into the fine-tuned second multimodal large model; The finely tuned second multimodal large model is used to perform traversal analysis on the image of information to be extracted, and to identify and output all information units in the document that match the type of the object to be extracted. If no information unit matching the extraction target type is detected, a feedback message indicating that the extraction target does not exist will be output. The general extraction instruction is used to instruct the analysis of the entire document content, without limiting the specific fields to be extracted.

14. A document classification and information extraction system based on a multimodal model, the system comprising: A processor and a memory for storing executable instructions; characterized in that the processor is configured to execute the executable instructions to perform the document classification and information extraction method based on a multimodal model as described in any one of claims 1 to 13.

15. A computer-readable storage medium, characterized in that, It stores a computer program, which, when executed by a processor, implements the document classification and information extraction method based on a multimodal model as described in any one of claims 1 to 13.

Images