End-to-end multi-modal information extraction method and device, electronic equipment and storage medium

Abstract

The application relates to the technical field of end-to-end multi-modal information extraction, and discloses an end-to-end multi-modal information extraction method and device, electronic equipment and a storage medium, wherein the method comprises the following steps: deep fusion of image visual features and text semantic features is carried out by using a visual language model, and a preset prompt word mechanism is used to accurately extract information for different document types, so that the accuracy is ensured, good generalization capability is maintained, and structured multi-modal information is directly output. The application has the beneficial effects that: the cumbersome secondary processing link is omitted, the automation degree and processing efficiency of document information extraction are greatly improved, the traditional multi-stage pipeline is integrated into a unified multi-modal reasoning process by constructing an end-to-end processing architecture, and the accumulation problem in the processing link of the prior art is effectively eliminated.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of end-to-end multi-modal information extraction, and particularly relates to an end-to-end multi-modal information extraction method and device, electronic equipment and a storage medium. BACKGROUND

[0002] With the deepening of the digitalization process, the financial, government, medical and other fields have accumulated a large amount of unstructured documents, such as contracts, bills and reports, and there is an urgent business demand to realize automatic information extraction. The existing technology generally adopts a multi-stage pipeline architecture of "Optical Character Recognition (OCR) -> text post-processing -> Natural Language Processing (NLP) extraction". This mode has inherent defects: first, the models of each stage are independent of each other, and the recognition errors of the upstream OCR link will be passed on and accumulated in the subsequent process, resulting in a decrease in the accuracy of the final result; in addition, the existing methods output pure text or non-standardized semi-structured results, which require a large amount of manual intervention for secondary sorting and standardization, and cannot directly generate machine-readable structured data such as JSON, which seriously restricts the efficiency and integration of the automation process. Therefore, an end-to-end solution is needed that can overcome the multi-stage processing errors and directly output standardized structured data. SUMMARY

[0003] Therefore, it is necessary to propose an end-to-end multi-modal information extraction method, device, electronic equipment and storage medium for the existing end-to-end multi-modal information extraction problem.

[0004] An end-to-end multi-modal information extraction method, the method comprising:

[0005] obtaining input document data; wherein the document data is in PDF or image format;

[0006] converting the document data into image data in a preset standard format;

[0007] identifying the image type of the image data;

[0008] obtaining the corresponding document prompt word according to the image type;

[0009] inputting the document prompt word and the image data into a preset visual language model to extract the target field corresponding to the document prompt word in the image data;

[0010] structurally processing the target field to obtain the first multi-modal information of the document data.

[0011] Further, the step of converting the document data into image data in a preset standard format comprises:

[0012] determining whether the document data is a PDF document;

[0013] if the document data is a PDF document, performing page segmentation on the document data and rendering each page as an image to obtain image document data in image format;

[0014] performing size normalization, resolution adjustment and color enhancement operations on the image document data to generate image data in a preset standard format that meets the input requirements of a visual language model.

[0015] Further, the step of obtaining a corresponding document prompt word according to the image type comprises:

[0016] determining whether the image type is in a preset pre-made type database;

[0017] if the image type is in the preset pre-made type database, obtaining a pre-made prompt word corresponding to the image type from the preset pre-made type database as the document prompt word;

[0018] if the image type is not in the preset pre-made type database, using a preset general prompt word as the document prompt word.

[0019] Further, the step of obtaining a pre-made prompt word corresponding to the image type from the preset pre-made type database as the document prompt word if the image type is in the preset pre-made type database comprises:

[0020] according to the identified image type, matching a basic prompt word template from the preset pre-made type database;

[0021] obtaining expected use information of the multi-modal information;

[0022] based on the expected use information, determining a set of extension fields that need to be extracted from the preset pre-made type database;

[0023] according to a predefined field priority rule, performing importance evaluation on the fields in the set of extension fields;

[0024] based on the evaluation result, dynamically supplementing description information of a target extension field with importance higher than a preset importance pre-made threshold to the basic prompt word template to obtain a pre-made prompt word and using the pre-made prompt word as the document prompt word.

[0025] Further, the step of dynamically supplementing description information of a target extension field with importance higher than a preset importance pre-made threshold to the basic prompt word template to obtain a pre-made prompt word and using the pre-made prompt word as the document prompt word based on the evaluation result comprises:

[0026] Based on the evaluation result, the target extension field with importance higher than the preset importance threshold is extracted;

[0027] A field description statement is generated for each target extension field; wherein the field description statement contains field definition, extraction range and format requirement;

[0028] According to the logical association between the field description statements, the field description statements are grouped and sorted;

[0029] The grouped and sorted field description statements are appended to the specified position of the basic prompt word template in natural language form to obtain a prefabricated prompt word and use it as a document prompt word.

[0030] Further, after the step of structurally processing the target field to obtain the first multi-modal information of the document data, the method further comprises:

[0031] On the basis of the basic prompt word, a variant prompt word is generated by synonym replacement or structure adjustment, and the visual language model is transformed or the document prompt word is transformed according to each of the variant prompt words to generate a plurality of second multi-modal information;

[0032] The first multi-modal information and the plurality of second multi-modal information are fused based on a predefined conflict resolution strategy to obtain third multi-modal information, and the third multi-modal information is used as the final multi-modal information of the document data, and the conflict resolution strategy includes: voting method, selection method based on confidence score, or weighted average method based on target field source reliability.

[0033] Further, the step of identifying the image type of the image data comprises:

[0034] The image data is input into a pre-trained document type classification model;

[0035] The document type classification model performs joint inference based on layout semantic features and visual entity features of the image data to output a document type probability distribution; wherein the layout semantic features are extracted by analyzing the spatial structure, region division and text line arrangement of the document; and the visual entity features are extracted by detecting whether there are logos, seals, specific table lines or handwritten notes in the document;

[0036] The image type of the image data is output according to the document type probability distribution.

[0037] An end-to-end multi-modal information extraction device, the device comprising:

[0038] A first acquisition module is configured to acquire input document data; wherein the document data is a PDF document or an image format document;

[0039] a conversion module, configured to convert the document data into image data in a preset standard format;

[0040] a recognition module, configured to recognize an image type of the image data;

[0041] a second acquisition module, configured to acquire a corresponding document prompt word according to the image type;

[0042] an input module, configured to input the document prompt word and the image data into a preset visual language model to extract a target field corresponding to the document prompt word in the image data;

[0043] a processing module, configured to perform structured processing on the target field to obtain first multi-modal information of the document data.

[0044] An electronic device, comprising a memory and a processor, the memory storing a computer program, the computer program being executed by the processor to cause the processor to perform the following steps:

[0045] acquire input document data; wherein the document data is a PDF document or an image format document;

[0046] convert the document data into image data in a preset standard format;

[0047] recognize an image type of the image data;

[0048] acquire a corresponding document prompt word according to the image type;

[0049] input the document prompt word and the image data into a preset visual language model to extract a target field corresponding to the document prompt word in the image data;

[0050] perform structured processing on the target field to obtain first multi-modal information of the document data.

[0051] A computer readable storage medium, storing a computer program, the computer program being executed by a processor to cause the processor to perform the following steps:

[0052] acquire input document data; wherein the document data is a PDF document or an image format document;

[0053] convert the document data into image data in a preset standard format;

[0054] recognize an image type of the image data;

[0055] acquire a corresponding document prompt word according to the image type;

[0056] inputting the document prompt word and the image data into a preset visual language model to extract a target field corresponding to the document prompt word in the image data;

[0057] performing structural processing on the target field to obtain first multi-modal information of the document data.

[0058] The present application has the beneficial effects that the visual language model is used to deeply fuse image visual features and text semantic features, the understanding and extraction accuracy of complex layout, table and picture-text mixed layout documents are significantly improved, the preset prompt word mechanism is used to accurately extract information for different document types, the generalization ability is maintained while the accuracy is ensured, the structured multi-modal information is directly output, the cumbersome secondary processing link is saved, the automation degree and processing efficiency of document information extraction are greatly improved, the traditional multi-stage pipeline is integrated into a unified multi-modal reasoning process through the construction of an end-to-end processing architecture, and the accumulation problem in the processing link of the prior art is effectively eliminated. BRIEF DESCRIPTION OF DRAWINGS

[0059] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0060] Among them:

[0061] Figure 1 It is an application environment diagram of the end-to-end multi-modal information extraction method in one embodiment;

[0062] Figure 2 It is a flowchart of the end-to-end multi-modal information extraction method in one embodiment;

[0063] Figure 3 It is a structural block diagram of the end-to-end multi-modal information extraction device in one embodiment;

[0064] Figure 4 It is a structural block diagram of an electronic device in one embodiment. DETAILED DESCRIPTION

[0065] With reference to the drawings of the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments of the present application, all the other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of the present application.

[0066] Figure 1 An application environment diagram of end-to-end multi-modal information extraction is provided in an embodiment. Referring to Figure 1 , the end-to-end multi-modal information extraction method is applied to an end-to-end multi-modal information extraction system. The end-to-end multi-modal information extraction system includes a terminal 110 and a server 120. The terminal 110 and the server 120 are connected through a network. The terminal 110 can be a desktop terminal or a mobile terminal. The mobile terminal can be at least one of a mobile phone, a tablet computer, a notebook computer, etc. The server 120 can be implemented by an independent server or a server cluster composed of multiple servers. The terminal 110 is used to obtain input document data, and the server 120 is used to extract multi-modal information of the document data.

[0067] As shown in Figure 2 , in an embodiment, an end-to-end multi-modal information extraction method is provided. The method can be applied to a terminal or a server. The embodiment is exemplified by application to a terminal. The end-to-end multi-modal information extraction method specifically includes the following steps:

[0068] S1: obtaining input document data; wherein the document data is in a PDF document or an image format document;

[0069] S2: converting the document data into image data in a preset standard format;

[0070] S3: identifying the image type of the image data;

[0071] S4: obtaining a corresponding document prompt word according to the image type;

[0072] S5: inputting the document prompt word and the image data into a preset visual language model to extract a target field corresponding to the document prompt word in the image data;

[0073] S6: performing structural processing on the target field to obtain first multi-modal information of the document data.

[0074] As described in step S1 above, the input document data is obtained, which can be in two main formats: PDF documents or image format documents. PDF (Portable Document Format) is a widely used file format that contains text, images, and graphical elements, commonly used for the storage and transmission of electronic documents. Image format documents usually refer to scanned image files (such as JPEG, PNG, etc.), which are derived from the scanning of paper documents and contain actual text and graphical information. The process of obtaining these document data involves file input interfaces or data upload functions, ensuring that users can easily select and import the documents to be processed.

[0075] As described in step S2 above, the document data is converted into image data in a preset standard format. For PDF documents, page processing is required to split them into individual pages and render each page as a high-quality image for subsequent feature extraction and information recognition. For image format documents, preprocessing operations such as image resizing, resolution optimization, and color enhancement are involved. These standardization steps aim to ensure the quality and format consistency of all input images to adapt to subsequent visual language models. The quality of the image directly affects the performance of the model, especially in complex document formats, where factors such as light, clarity, and contrast can affect recognition results. Therefore, ensuring standardized image data processing can improve accuracy and reduce errors in the information extraction stage, significantly optimizing the efficiency and effectiveness of overall information extraction. The preset standard format can be an image with a resolution of ≥300 DPI and an RGB color mode.

[0076] As described in step S3 above, the image type of the image data is identified. Image type identification will be based on feature analysis and machine learning techniques, specifically through a trained classification model to identify the specific type of document, such as identifying whether the document is an invoice, contract, receipt, or other text-based file. The model will extract visual features from the image, analyze the document's layout, text layout, and visual elements, and thus effectively classify it. For different types of documents, their content, structure, and important fields may vary greatly, so quickly and accurately identifying the image type is a core pre-step for information extraction.

[0077] As described in S4 above, the document prompt words adapted to the type of the image just identified are obtained according to the type of the image just identified, and the prompt words are used to guide the subsequent information extraction process, ensuring that the model focuses on the main fields that need to be extracted in the document, for example, for an invoice type document, the fields such as "invoice number", "invoice date", "seller name" need to be focused on; while for a contract type document, the fields such as "contract number", "signing date", "party A and party B information" need to be extracted. The prompt word information can be saved in a keyword library with a predefined structure, and the corresponding prompt word template can be quickly retrieved from the keyword library according to the specific image type. At the same time, the prompt words can be designed to contain structural information of information extraction rules and field format requirements, helping the model to follow certain semantic and format requirements in the extraction process, and ensuring that the output structured data meets the expectations.

[0078] As described in step S5 above, the document prompt words and the image data are input into a preset visual language model to extract the target fields in the image data corresponding to the document prompt words. The obtained document prompt words and image data are input into a pre-trained visual language model (VLM) which integrates the understanding ability of visual features and text information, and can accurately identify the target field information through an efficient cross-modal learning process. Specifically, the model dynamically allocates weights through an attention mechanism, combines visual information with text prompt words, and identifies the text content related to the prompt words and its layout and position in the image. The advantage of this process is that the model not only relies on text information, but also integrates the spatial features of the image, thereby improving the understanding ability of complex documents. The target fields extracted by the model correspond to the input prompt words, which can ensure the completeness and accuracy of the extraction of key information.

[0079] As described in step S6 above, the target fields are structured to obtain the first multi-modal information of the document data. The extracted target fields are structured to generate the target multi-modal information. Structuring means converting the extracted raw data into a format that can be directly processed and understood by a computer, such as JSON or other standardized structures. This process includes several key sub-steps: first, the field information needs to be cleaned, including removing redundant characters, ensuring field format consistency (such as date, amount, etc.), and unifying the case of the text. Then, field mapping is performed, which maps the extracted field names to predefined standard field names to ensure consistency and standardization of the output results. Finally, since the structured information of the target fields may have a significant impact on fast and efficient data access and subsequent processing, certain format rules need to be followed. Through this series of processing, the finally generated multi-modal information not only contains complete document key information, but also is easy to use in subsequent use, which can effectively reduce the secondary processing cost of data analysis and provide accurate basis for immediate decision-making.

[0080] In one embodiment, the step S2 of converting the document data into image data in a preset standard format comprises:

[0081] S201: determining whether the document data is a PDF document;

[0082] S202: if the document data is a PDF document, performing page processing on the document data, and rendering each page into an image, thereby obtaining image document data in image format;

[0083] S203: performing size normalization, resolution adjustment and color enhancement operations on the image document data to generate image data in a preset standard format that meets the input requirements of the visual language model.

[0084] As described in step S201 above, it is determined whether the document data is a PDF document. This determination can be based on the inspection of the file extension or the analysis of the file header information. PDF (Portable Document Format) is a widely used file format that can accommodate nested structured information such as text, graphics, images, etc. The necessity of determining the document type lies in the fact that different formats of documents will adopt different processing strategies in the processing flow. If the result of the determination is a PDF document, the system needs to be processed according to specific rules; if it is a file of other types (such as image formats such as JPEG, PNG, etc.), different processing logic needs to be used.

[0085] As described in step S202 above, if the document data is a PDF document, the document data is processed by page, and each page is rendered into an image, thereby obtaining image document data in image format. After confirming that the document data is a PDF document, page processing is started, which specifically involves splitting the PDF document by page and processing each page independently. This process utilizes a special PDF processing library (such as PyMuPDF or pdf2image) that can parse PDF files and convert each page into a high-quality image. The rendering of each page should ensure the clarity of the image data, especially when performing subsequent information extraction, as the image quality directly affects the accuracy of OCR and subsequent domain model inference. During the page processing, the system retains the page number information to enable tracing to the corresponding document location during subsequent processing. In this way, the image data of each page obtained by the user can be integrated or analyzed individually, laying a good foundation for the processing of multi-modal information.

[0086] As described in step S203 above, size normalization, resolution adjustment, and color enhancement operations (e.g., adjusting image contrast using histogram equalization) are performed on the image document data to generate image data in a preset standard format that meets the input requirements of the visual language model. A series of processes are performed on the rendered image document data to ensure that it meets the standard format for input into the visual language model. First, size normalization refers to adjusting the size of each image to a preset uniform specification, which helps to reduce instability caused by different image sizes when inputting the model, ensuring that the model can effectively learn and reason under consistent input conditions. Next, resolution adjustment aims to improve image clarity, typically adjusting the image resolution to a standard suitable for processing, such as 300 DPI (dots per inch). Finally, color enhancement operations improve the visual quality of the image to enhance the readability of key content, such as through contrast, brightness settings, gamma correction, or color saturation adjustments, ensuring that key information is more prominent and easily identifiable. Proper execution of these processing steps lays a foundation for subsequent information extraction and model reasoning, improving the accuracy and reliability of the entire multi-modal information processing process, while also clearly presenting potential visual features to the visual language model, allowing the model to more perfectly capture and extract the required information.

[0087] In one embodiment, the step S4 of obtaining the corresponding document prompt word according to the image type comprises:

[0088] S401: determining whether the image type is in a preset pre-made type database;

[0089] S402: if it is in the preset pre-made type database, obtaining the pre-made prompt word corresponding to the image type from the preset pre-made type database as the document prompt word;

[0090] S403: if it is not in the preset pre-made type database, using a preset general prompt word as the document prompt word.

[0091] As described in step S401 above, it is checked whether the image type obtained through image recognition exists in a preset pre-made type database defined in advance, which contains classification information of various common document types, such as invoices, contracts, receipts, and application forms. The judgment process is usually through finding the label or attribute of the image type and comparing whether it matches the document types defined in the database, ensuring that the extracted prompt word can accurately correspond to a specific document category, thereby improving the accuracy of subsequent information extraction. If the recognized image type is in the database, it means that the document is typical and can be applied to known extraction strategies. Conversely, if the image type is not recognized, the system needs to use a general prompt word, and the extraction efficiency and accuracy using a general prompt word will decrease to some extent.

[0092] As described in step S402 above, if it is determined that the image type does indeed exist in the preset pre-made type database, then prompt words are obtained. This process involves querying the database for pre-made prompt words that match the identified image type. These pre-made prompt words are specifically customized for different document types, aiming to provide clear guidance and definitions of key fields for the information extraction process. For example, for invoice-type documents, pre-made prompt words may include "invoice number," "invoice date," "buyer information," etc., along with corresponding formats and extraction guidelines. In this way, the system can ensure the accuracy of information extraction and reduce erroneous extraction caused by ambiguous guidance. This provides a basic framework for subsequent multimodal information extraction, enabling the model to clearly understand the definition and function of target fields, thereby achieving accurate and stable information extraction results.

[0093] As described in step S403 above, if it is determined that the identified image type is not in the preset pre-built type database, a preset universal prompt word will be used as the document prompt word. The universal prompt word is an abstract guide designed for handling unseen or unknown document types. It is applicable to various document formats. These prompt words are usually quite general. For example, they may require the extraction of information such as "date", "amount", "name", and "number". This ensures that information extraction can be performed when handling uncommon or specific document types, while retaining a certain degree of flexibility and adaptability.

[0094] In one embodiment, step S402, which involves retrieving a pre-defined prompt word corresponding to the image type from the preset pre-defined type database as a document prompt word if the image type is in the preset pre-defined type database, includes:

[0095] S4021: Based on the identified image type, match a basic prompt word template from the preset type database;

[0096] S4022: Obtain the intended use information of the multimodal information;

[0097] S4023: Based on the intended use information, determine the set of extended fields to be extracted from the preset prefabricated type database;

[0098] S4024: Based on predefined field priority rules, evaluate the importance of the fields in the extended field set;

[0099] S4025: Based on the evaluation results, the descriptive information of the target extended fields with importance higher than the preset importance threshold is dynamically added to the basic prompt word template to obtain the pre-made prompt words and use them as document prompt words.

[0100] As described in step S4021 above, based on the previously identified image type, the system queries and matches the corresponding basic prompt word template from a pre-set type database. The basic prompt word template is specifically designed for different types of documents (e.g., invoices, contracts, certificates, etc.) to provide a basic framework for information extraction. This process is typically achieved by searching for relevant keywords or tags; that is, the system selects the most suitable template based on the image's features (e.g., structure, format, etc.). The advantage of matching the basic prompt word template is that it provides a standardized description method, ensuring that subsequent extraction processes are conducted within a consistent framework, thereby improving the accuracy and consistency of information extraction. This allows subsequent prompt word generation to start efficiently, facilitating rapid data extraction for specific document types.

[0101] As described in step S4022 above, information regarding the intended use of the multimodal information is obtained. This intended use information refers to user-inputted or preset functional requirements, describing the scenarios or purposes for which the extracted information will be used. For example, for financial auditing, invoice information can be extracted; for legal compliance, the terms and conditions of contracts need to be extracted. By analyzing the intended use, a more suitable set of fields can be selected in a targeted manner, providing a clear direction for subsequent data processing. This targeted design effectively avoids invalid or duplicate data extraction caused by unclear information extraction direction, improving the information processing efficiency of the entire system. The intended use information can be obtained through user interface input or configuration in the system backend to ensure flexible response to various document processing needs.

[0102] As described in step S4023 above, based on the previously acquired intended use information, a set of extended fields to be extracted is determined from a pre-defined pre-built type database. The set of extended fields varies depending on the intended use and may include multiple fields directly related to the document content, such as "invoice issuer," "transaction amount," and "product details" in an invoice, or "signature information of all parties," "signing date," and "effective terms" in a contract. This process typically relies on a pre-built purpose-field mapping table, which clarifies the association between different document types and purposes. By accurately extracting the required fields, redundant data extraction can be avoided, thereby optimizing information flow early and improving the focus and relevance of the extraction task.

[0103] As described in step S4024 above, the importance of each field in the extended field set is evaluated according to predefined field priority rules. Field priority rules may be based on multiple factors. For example, in certain application scenarios, specific fields may be crucial to business decisions; for instance, the "amount" field has a higher priority than other fields in invoice processing. Furthermore, the evaluation rules may also consider field frequency, legal requirements, and the accuracy of historical data extraction. For example, fields such as "signature" in a certain document type may be particularly important in some scenarios, while less critical in others. Through such evaluation, the system can score or rate the priority of each field, thereby focusing on those highly important fields during information extraction, optimizing the efficiency and quality of data extraction. This step not only helps reduce the overhead of processing unnecessary data but also ensures that, with limited resources, key business needs can be prioritized, enhancing the system's practical value and flexibility.

[0104] As described in step S4025 above, based on the evaluation results, the descriptive information of target extended fields whose importance is higher than the preset importance threshold is dynamically added to the basic prompt word template to obtain the pre-defined prompt words and use them as document prompt words. That is, according to the importance evaluation results of the previous step, the descriptive information of target extended fields whose importance is higher than the preset threshold is dynamically added to the basic prompt word template. The specific method of dynamic addition can use a template filling algorithm to fill the field definition, extraction range and format requirements into the basic prompt word template. The preset importance threshold is a configurable numerical threshold used to filter field priority. The threshold range that the user can configure is (0-1) to filter high-importance fields. Specifically, the evaluated extended fields are filtered to identify those marked as highly important. The descriptive information of these fields (such as their meaning and extraction requirements) is integrated and embedded into the previously selected basic prompt template. The resulting pre-generated prompts not only contain basic information but also necessary external extended information, ensuring the comprehensiveness and accuracy of information extraction. Through this dynamically supplemented mechanism, the system can efficiently handle increasingly complex document processing scenarios, ensuring that information extraction targets are clear and targeted. The final generated document prompts will not merely be simple instructions but rich and structured information guidance, improving the overall intelligence level of document data extraction.

[0105] In one embodiment, step S4025, which dynamically supplements the description information of target extended fields with importance higher than a preset importance threshold into the basic prompt word template based on the evaluation results to obtain pre-set prompt words and use them as document prompt words, includes:

[0106] S40251: Based on the evaluation results, extract target extended fields whose importance is higher than the preset importance threshold;

[0107] S40252: Generate a field description statement for each target extended field; wherein, the field description statement includes field definition, extraction scope and format requirements;

[0108] S40253: Group and sort the field description statements according to the logical relationship between them;

[0109] S40254: The grouped and sorted field description statements are appended to the specified position of the basic prompt word template in natural language form to obtain a pre-made prompt word and use it as a document prompt word.

[0110] As described in step S40251 above, target extended fields with importance assessment values ​​higher than the preset importance threshold are selected based on the priority assessment results of the previous fields. This process relies on the previously set threshold standard. The preset threshold is a quantitative indicator set for specific application scenarios. Its purpose is to accurately define which fields are worth prioritizing in a specific document or purpose. For example, for a certain document type, the importance threshold for the field "invoice amount" is set to 0.7. If the field's assessment score is 0.8, the system will mark the field as a target extended field and proceed with subsequent processing. Through this mechanism, the system can effectively concentrate resources on important fields, enhance the efficiency of information extraction, and thus not only ensure the quality of information extraction but also provide clear guidance for the subsequently generated prompts, ensuring that the final output meets the user's needs and business scenarios, and reducing unnecessary computation and storage overhead.

[0111] As described in step S40252 above, a professional field description statement is generated for each element identified as a target extended field. The field description statement provides the necessary context and specific guidance for information extraction. Each description statement typically includes three core components: field definition, extraction scope, and format requirements. The field definition uses simple and clear language to explain the specific meaning of the field, such as "invoice amount refers to the total cost of the transaction," ensuring the model understands the nature of the field. The extraction scope specifies the specific location or context of the field in the document, helping the model better locate relevant information, such as "the total amount field at the bottom of the document." The format requirements define the structure that the field data should follow, such as numeric formats (e.g., "retain two decimal places"), date format requirements (e.g., "YYYY-MM-DD"), etc.

[0112] As described in step S40253 above, the field description statements are grouped and sorted according to the logical relationships between them. Reasonable grouping and sorting helps improve the efficiency and accuracy of information extraction. By analyzing keywords, semantic relationships, and interdependencies in the description statements, it is possible to identify which fields are related to or dependent on other fields. For example, if "total invoice amount" depends on the sum of various fees in "product details", the system will put these related fields in the same group to ensure that they can be processed simultaneously during information extraction. In addition, the field description statements can be sorted according to importance, the required extraction time order, or logical relationships to ensure that the most critical fields are processed first during extraction. Such grouping and sorting not only provides a good logical structure for information extraction but also promotes the model to follow a more natural decision chain during inference, thereby improving the overall extraction effect and user experience.

[0113] As described in step S40254 above, the grouped and sorted field description statements are appended to the designated positions of the basic prompt word template in natural language form to obtain pre-made prompt words, which are then used as document prompt words. Under the premise of ensuring semantic coherence and logical rationality, the grouped and sorted field descriptions are filled into the designated positions of the template according to the prescribed structure, thereby forming complete pre-made prompt words. These prompt words not only contain the original basic information but also incorporate relevant field description information, enabling the newly generated prompt words to provide clearer guidance and direction, helping the subsequent visual language model to accurately identify and extract important fields. Through this supplementary operation, the generated document prompt words become more information-rich and have stronger adaptability and targeting.

[0114] In one embodiment, after step S6 of performing structured processing on the target field to obtain the first multimodal information of the document data, the method further includes:

[0115] S701: Based on the basic prompt words, variant prompt words are generated by synonym replacement or structural adjustment, the visual language model is transformed, or the document prompt words are transformed sequentially according to each of the variant prompt words to generate multiple corresponding second multimodal information;

[0116] S702: The first multimodal information and multiple second multimodal information are fused based on a predefined conflict resolution strategy to obtain third multimodal information, and the third multimodal information is used as the final multimodal information of the document data. The conflict resolution strategy includes: voting method, selection method based on confidence score, or weighted average method based on the reliability of the source of the target field.

[0117] As described in step S701 above, variant prompts are generated based on the basic prompts through synonym replacement or structural adjustment. The visual language model is transformed, or the document prompts are transformed sequentially according to each variant prompt, to generate multiple corresponding second multimodal information. Multiple transformation operations are performed on the document prompts or visual language model to generate multiple different second multimodal information. This transformation can take various forms. On the one hand, multiple variants can be formed by fine-tuning or combining the basic prompts to adapt to different extraction needs or scenarios. On the other hand, the visual language model itself can be trained or optimized multiple times so that it can generate multiple different output results for the same input image data. Diverse outputs help ensure that key field information is mined and covered to the greatest extent possible during the information extraction process. The richness and depth of the final information can be increased by exploring different extraction methods, enabling the model to understand and learn the same data from multiple perspectives, enhancing the comprehensiveness and accuracy of the information. This multiple generation process helps to improve the quality of the final output.

[0118] As described in step S702 above, the first multimodal information and multiple second multimodal information are fused based on a predefined conflict resolution strategy to obtain third multimodal information, which is then used as the final multimodal information of the document data. The conflict resolution strategy includes: a voting method, a selection method based on confidence scores, or a weighted average method based on the reliability of the target field source. To effectively handle potentially contradictory or conflicting information from different sources, a predefined conflict resolution strategy is employed. In practice, conflict resolution strategies can be tailored to specific circumstances. For example, voting is suitable for selecting from multiple outputs, determining the final selected field based on the repeated extraction of each field. A confidence-based selection method assesses the reliability of each information fragment, prioritizing higher-scoring information in the final output. A weighted average method based on the source reliability of the target field assigns greater weight to highly reliable information based on the source's trust level, thereby reducing the impact of low-quality information on the final result during fusion. The aim of this process is to fully utilize the diversity provided by different information sources while maintaining high accuracy and consistency, enhancing the multimodal information assurance of the final document, and thus improving user trust and effectiveness in data processing.

[0119] In one embodiment, step S3, which identifies the image type of the image data, includes:

[0120] S301: Input the image data into a pre-trained document type classification model;

[0121] S302: The document type classification model performs joint reasoning based on the layout semantic features and visual entity features of the image data, and outputs a document type probability distribution; wherein, the layout semantic features are extracted by analyzing the spatial structure, region division and text line arrangement of the document; the visual entity features are extracted by detecting whether there are logos, seals, specific table lines or handwriting in the document;

[0122] S303: Output the image type of the image data according to the document type probability distribution.

[0123] As described in step S301 above, the processed image data is input into a pre-trained document type classification model. This model is typically a deep learning model trained with a large amount of example document data, with the aim of enabling it to recognize different document types. The pre-trained model can effectively classify new and unseen documents based on the learned features. The input image data can be in the form of images of various documents, such as invoices, contracts, receipts, etc. At the beginning of the processing flow, the system needs to ensure that the input image data matches the model's expected input format, such as size, resolution, and number of channels. This can prevent potential input errors from affecting the classification results. By transmitting the image data to the classification model, the system initiates intelligent analysis capabilities based on image content, providing a technical foundation for subsequent document type recognition.

[0124] As described in step S302 above, the document type classification model analyzes the layout semantic features and visual entity features of the input image data for joint inference. Layout semantic features are mainly extracted by analyzing the spatial structure, region division (such as titles, paragraphs, tables, etc.), and text layout information of the document content. For example, in an invoice, the invoice number is usually located in the upper right corner of the document and has a specific layout relationship with other content. Visual entity features are extracted by detecting specific items in the document (such as logos, stamps, specific table lines, and handwriting). These visual elements not only help identify the document type but are also important indicators of the document's authenticity and validity. By combining these two features, the model can more comprehensively understand the document content and thus make a more accurate classification judgment.

[0125] As described in step S303 above, the image type of the image data is output based on the document type probability distribution. The specific type of the image data is determined and output based on the probability distribution generated by the document type classification model. The output probability distribution represents the model's confidence level in all possible document types, usually expressed as a percentage. The sum of the probability vectors for each document type output by the document type classification model is 1. For example, the model might determine that a certain image data has an 80% probability of being an invoice, a 15% probability of being a contract, and a 5% probability of being a receipt. To determine the final image type, the system sets a predefined threshold to determine which type has a sufficiently high probability to be accepted as the most likely result. If the predicted probability of a certain document type exceeds the set threshold, the system will output that type as the final result. For example, when the predicted probability of an invoice is significantly higher than that of a contract and a receipt, the identification result is determined to be an invoice. Ultimately, effectively converting the image data type into semantic information usable by the system is a key step in achieving accurate and efficient information extraction.

[0126] Reference Figure 3 The present invention also provides an end-to-end multimodal information extraction device, the device comprising:

[0127] The first acquisition module 902 is used to acquire input document data; wherein, the document data is a PDF document or an image format document;

[0128] The conversion module 904 is used to convert the document data into image data in a preset standard format;

[0129] The recognition module 906 is used to identify the image type of the image data;

[0130] The second acquisition module 908 is used to acquire the corresponding document prompt words according to the image type;

[0131] The input module 910 is used to input the document prompt words and the image data into a preset visual language model in order to extract the target field corresponding to the document prompt words in the image data;

[0132] The processing module 912 is used to perform structured processing on the target field to obtain the first multimodal information of the document data.

[0133] In one embodiment, the conversion module 904 includes:

[0134] The document data determination submodule is used to determine whether the document data is a PDF document;

[0135] The pagination processing submodule is used to perform pagination processing on the document data when the document data is a PDF document, and render each page as an image to obtain image document data in image format;

[0136] The normalization processing submodule is used to perform size normalization, resolution adjustment and color enhancement operations on the image document data to generate image data in a preset standard format that meets the input requirements of the visual language model.

[0137] In one embodiment, the second acquisition module 908 includes:

[0138] The image type determination submodule is used to determine whether the image type is in a preset type database;

[0139] The pre-defined prompt word acquisition submodule is used to retrieve the pre-defined prompt word corresponding to the image type from the preset pre-defined type database if the image type is in the preset pre-defined type database, and use it as the document prompt word.

[0140] The document prompt tagging submodule is used to use a preset general prompt as the document prompt if it is not in the preset pre-built type database.

[0141] In one embodiment, the pre-set prompt word acquisition submodule includes:

[0142] The matching unit is used to match a basic prompt word template from the preset type database based on the identified image type;

[0143] The acquisition unit is used to acquire the intended use information of the multimodal information;

[0144] The determining unit is used to determine the set of extended fields to be extracted from the preset prefabricated type database based on the intended use information.

[0145] An evaluation unit is used to evaluate the importance of fields in the extended field set according to predefined field priority rules;

[0146] The supplementary unit is used to dynamically supplement the description information of target extended fields with importance higher than a preset importance threshold into the basic prompt word template based on the evaluation results, so as to obtain the pre-made prompt words and use them as document prompt words.

[0147] In one embodiment, the supplementary unit includes:

[0148] Extract sub-units, used to extract target extended fields with importance higher than a preset importance threshold based on the evaluation results;

[0149] A generation subunit is used to generate a field description statement for each target extended field; wherein, the field description statement includes field definition, extraction scope and format requirements;

[0150] The sorting subunit is used to group and sort the field description statements according to the logical relationship between them;

[0151] An append subunit is used to append the grouped and sorted field description statements in natural language to a specified position in the basic prompt word template to obtain a pre-made prompt word and use it as a document prompt word.

[0152] In one embodiment, the end-to-end multimodal information extraction device further includes:

[0153] The document prompt word transformation module is used to generate variant prompt words based on the basic prompt words by replacing synonyms or adjusting the structure, transform the visual language model, or sequentially transform the document prompt words according to each of the variant prompt words to generate corresponding multiple second multimodal information.

[0154] The fusion module is used to fuse the first multimodal information and multiple second multimodal information based on a predefined conflict resolution strategy to obtain third multimodal information, and to use the third multimodal information as the final multimodal information of the document data. The conflict resolution strategy includes: voting method, selection method based on confidence score, or weighted average method based on the reliability of the source of the target field.

[0155] In one embodiment, the identification module 906 includes:

[0156] The image data input submodule is used to input the image data into a pre-trained document type classification model;

[0157] The reasoning submodule is used by the document type classification model to perform joint reasoning based on the layout semantic features and visual entity features of the image data, and output the document type probability distribution; wherein, the layout semantic features are extracted by analyzing the spatial structure, region division and text line arrangement of the document; the visual entity features are extracted by detecting whether there are logos, seals, specific table lines or handwriting in the document;

[0158] The image type output submodule is used to output the image type of the image data according to the document type probability distribution.

[0159] Figure 4 An internal structural diagram of an electronic device in one embodiment is shown. This electronic device can specifically be a terminal or a server, and more specifically, a computer device. Figure 4As shown, the electronic device includes a processor, a memory, and a network interface connected via a system bus. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and may also store a computer program. When executed by the processor, this computer program enables the processor to implement an end-to-end multimodal information extraction method. The internal memory may also store a computer program, which, when executed by the processor, enables the processor to implement the end-to-end multimodal information extraction method. Those skilled in the art will understand that… Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the electronic device to which the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0160] In one embodiment, an electronic device is provided, including a memory and a processor, the memory storing a computer program that, when executed by the processor, causes the processor to perform the following steps:

[0161] Obtain the input document data; wherein the document data is a PDF document or an image format document;

[0162] The document data is converted into image data in a preset standard format;

[0163] Identify the image type of the image data;

[0164] Obtain the corresponding document prompt words based on the image type;

[0165] The document prompt and the image data are input into a preset visual language model to extract the target field corresponding to the document prompt in the image data;

[0166] The target field is subjected to structured processing to obtain the first multimodal information of the document data.

[0167] By leveraging a visual language model to deeply integrate image visual features and text semantic features, the understanding and extraction accuracy of complex layouts, tables, and mixed text and image documents are significantly improved. Through a pre-set prompt word mechanism, precise information extraction can be performed for different document types, maintaining good generalization ability while ensuring accuracy. Finally, structured multimodal information is directly output, eliminating cumbersome secondary processing steps and greatly improving the automation and efficiency of document information extraction. By constructing an end-to-end processing architecture, the traditional multi-stage pipeline is integrated into a unified multimodal reasoning process, effectively eliminating the accumulated problems in the processing stages of existing technologies.

[0168] In one embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, causes the processor to perform the following steps:

[0169] Obtain the input document data; wherein the document data is a PDF document or an image format document;

[0170] The document data is converted into image data in a preset standard format;

[0171] Identify the image type of the image data;

[0172] Obtain the corresponding document prompt words based on the image type;

[0173] The document prompt and the image data are input into a preset visual language model to extract the target field corresponding to the document prompt in the image data;

[0174] The target field is subjected to structured processing to obtain the first multimodal information of the document data.

[0175] By leveraging a visual language model to deeply integrate image visual features and text semantic features, the understanding and extraction accuracy of complex layouts, tables, and mixed text and image documents are significantly improved. Through a pre-set prompt word mechanism, precise information extraction can be performed for different document types, maintaining good generalization ability while ensuring accuracy. Finally, structured multimodal information is directly output, eliminating cumbersome secondary processing steps and greatly improving the automation and efficiency of document information extraction. By constructing an end-to-end processing architecture, the traditional multi-stage pipeline is integrated into a unified multimodal reasoning process, effectively eliminating the accumulated problems in the processing stages of existing technologies.

[0176] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0177] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0178] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. An end-to-end multimodal information extraction method, characterized in that, The method includes: Obtain the input document data; wherein the document data is a PDF document or an image format document; The document data is converted into image data in a preset standard format; Identify the image type of the image data; Obtain the corresponding document prompt words based on the image type; The document prompt and the image data are input into a preset visual language model to extract the target field corresponding to the document prompt in the image data; The target field is subjected to structured processing to obtain the first multimodal information of the document data; The step of obtaining the corresponding document prompt words based on the image type includes: Determine whether the image type is in a preset type database; If it is in the preset type database, then a basic prompt word template is matched from the preset type database according to the identified image type; Obtain the intended use information of the multimodal information; Based on the intended use information, determine the set of extended fields to be extracted from the preset prefabricated type database; The importance of the fields in the extended field set is evaluated according to predefined field priority rules; Based on the evaluation results, target extended fields with importance higher than the preset importance threshold are extracted; Generate a field description statement for each target extended field; wherein the field description statement includes the field definition, extraction scope and format requirements; Based on the logical relationships between the field description statements, the field description statements are grouped and sorted. The grouped and sorted field description statements are appended in natural language to the specified position of the basic prompt word template to obtain pre-made prompt words, which are then used as document prompt words. The step of identifying the image type of the image data includes: The image data is input into a pre-trained document type classification model; The document type classification model performs joint inference based on the layout semantic features and visual entity features of the image data, and outputs a document type probability distribution; wherein, the layout semantic features are extracted by analyzing the spatial structure, region division and text line arrangement of the document; the visual entity features are extracted by detecting whether there are logos, seals, specific table lines or handwriting in the document; The image type of the image data is output based on the document type probability distribution.

2. The end-to-end multimodal information extraction method according to claim 1, characterized in that, The step of converting the document data into image data in a preset standard format includes: Determine whether the document data is a PDF document; If the document data is a PDF document, the document data is paginated and each page is rendered as an image to obtain image document data in image format; The image document data is subjected to size normalization, resolution adjustment, and color enhancement operations to generate image data in a preset standard format that meets the input requirements of the visual language model.

3. The end-to-end multimodal information extraction method according to claim 1, characterized in that, After the step of performing structured processing on the target field to obtain the first multimodal information of the document data, the method further includes: Based on the basic prompt words, variant prompt words are generated by synonym replacement or structural adjustment, the visual language model is transformed, or the document prompt words are transformed sequentially according to each of the variant prompt words to generate multiple corresponding second multimodal information; The first multimodal information and multiple second multimodal information are fused based on a predefined conflict resolution strategy to obtain third multimodal information, and the third multimodal information is used as the final multimodal information of the document data. The conflict resolution strategy includes: voting method, selection method based on confidence score, or weighted average method based on the reliability of the source of the target field.

4. An end-to-end multimodal information extraction device, characterized in that, The device includes: The first acquisition module is used to acquire input document data; wherein, the document data is a PDF document or an image format document; The conversion module is used to convert the document data into image data in a preset standard format; The recognition module is used to identify the image type of the image data; The second acquisition module is used to acquire the corresponding document prompt words according to the image type; An input module is used to input the document prompt and the image data into a preset visual language model to extract the target field corresponding to the document prompt in the image data; The processing module is used to perform structured processing on the target field to obtain the first multimodal information of the document data; The second acquisition module includes: The image type determination submodule is used to determine whether the image type is in a preset type database; The pre-defined prompt word acquisition submodule is used to retrieve the pre-defined prompt word corresponding to the image type from the preset pre-defined type database if the image type is in the preset pre-defined type database, and use it as the document prompt word. The document prompt word tagging submodule is used to use a preset general prompt word as the document prompt word if it is not in the preset pre-made type database; The pre-defined prompt word acquisition submodule includes: The matching unit is used to match a basic prompt word template from the preset type database based on the identified image type; The acquisition unit is used to acquire the intended use information of the multimodal information; The determining unit is used to determine the set of extended fields to be extracted from the preset prefabricated type database based on the intended use information. An evaluation unit is used to evaluate the importance of fields in the extended field set according to predefined field priority rules; The supplementary unit is used to dynamically supplement the description information of the target extended field with an importance higher than the preset importance threshold into the basic prompt word template based on the evaluation results, so as to obtain the pre-made prompt words and use them as document prompt words; The supplementary unit includes: Extract sub-units, used to extract target extended fields with importance higher than a preset importance threshold based on the evaluation results; A generation subunit is used to generate a field description statement for each target extended field; wherein, the field description statement includes field definition, extraction scope and format requirements; The sorting subunit is used to group and sort the field description statements according to the logical relationship between them; An append subunit is used to append the grouped and sorted field description statements in natural language to a specified position in the basic prompt word template to obtain pre-made prompt words and use them as document prompt words; The identification module includes: The image data input submodule is used to input the image data into a pre-trained document type classification model; The reasoning submodule is used by the document type classification model to perform joint reasoning based on the layout semantic features and visual entity features of the image data, and output the document type probability distribution; wherein, the layout semantic features are extracted by analyzing the spatial structure, region division and text line arrangement of the document; the visual entity features are extracted by detecting whether there are logos, seals, specific table lines or handwriting in the document; The image type output submodule is used to output the image type of the image data according to the document type probability distribution.

5. A computer-readable storage medium, characterized in that, The system contains a computer program that, when executed by a processor, causes the processor to perform the steps of the end-to-end multimodal information extraction method as described in any one of claims 1 to 3.

6. An electronic device, characterized in that, The device includes a memory and a processor, the memory storing a computer program that, when executed by the processor, causes the processor to perform the steps of the end-to-end multimodal information extraction method as described in any one of claims 1 to 3.

Images