Intelligent extraction method, system and device for complex document based on multi-modal large model and storage medium

Abstract

The application provides a complex document intelligent extraction method, system and device based on a multi-modal large model and a storage medium, and relates to the technical field of automatic data processing and information extraction. The disclosed technology provides an efficient, accurate and self-optimizing information extraction solution, which includes a combination of technologies: a two-stage extraction strategy using preliminary screening and deep information extraction balances the efficiency and depth of information extraction; a cross-page dependency graph construction mechanism solves the problem of information fragmentation; a context-aware overall information extraction method ensures that the model reasons in a complete visual context; dynamic task decomposition and dimension reduction techniques are also used to address performance challenges caused by complex tasks; and a prompt word self-optimization closed loop based on artificial feedback is used to achieve continuous learning and precision improvement of the system.

Description

Technical Field

[0001] This invention relates to the field of automated data processing and information extraction technology, specifically to a method, system, device, and storage medium for intelligent extraction of complex documents based on a multimodal large model. Background Technology

[0002] In modern medical practice, clinical practice guidelines are crucial for clinical treatment decisions. This is especially true in the field of oncology, where the diversity of cancer types and the rapid pace of advancements in cancer treatment research result in oncology clinical practice guidelines characterized by their large volume of information and frequent updates. For example, guidelines published by the National Comprehensive Cancer Network (NCCN) are widely recognized as authoritative in the field of oncology treatment, currently covering over 60 cancer types, with most updated several times a year. However, up to 80% of the critical medical information in these clinical practice guidelines exists in unstructured or semi-structured free text formats. This forces clinicians to spend considerable time manually searching and extracting information, a process that is not only inefficient but also prone to errors, severely hindering the timely and accurate application of the latest and best clinical practices.

[0003] To address the difficulties and problems faced by clinicians in manually searching for information, the industry has developed various information extraction technologies, but all have significant limitations. First, traditional OCR technology often fails when processing complex medical documents, exhibiting fundamental shortcomings: for example, it struggles to handle information loss or misalignment due to complex layouts, lacks contextual understanding to distinguish information types and related data, is highly dependent on image quality, and fundamentally loses non-textual information during conversion. Second, even if perfect text can be obtained through some means, relying solely on Natural Language Processing (NLP) technologies faces significant challenges: the specialized complexity of medical language makes it difficult for general NLP models to handle, and existing NLP systems cannot effectively integrate information fragmentation and cross-page references found in medical guidelines.

[0004] Based on this, this application proposes a novel technical solution in order to solve one or more of the aforementioned technical problems. Summary of the Invention

[0005] (a) Technical problems to be solved

[0006] To address the shortcomings of existing technologies, this invention provides a method, system, device, and storage medium for intelligent extraction of complex documents based on a multimodal large model, solving one or more of the aforementioned technical problems.

[0007] (II) Technical Solution

[0008] To achieve the above objectives, the present invention provides the following technical solution:

[0009] Firstly, this application proposes a method for intelligent extraction of complex documents based on a multimodal large model, the method comprising:

[0010] The complex document to be extracted is converted into a set of page images according to the predetermined requirements;

[0011] The set of page images is initially filtered based on a multimodal large model to obtain a set of related pages;

[0012] Construct a cross-page dependency graph based on the aforementioned set of related pages;

[0013] For the extraction task to be executed, the multimodal large model is used again to extract deep information from the dependency graph related to the extraction task.

[0014] In one embodiment, the page image set is initially filtered based on a multimodal large model to obtain a set of associated pages, including:

[0015] By using a multimodal large model to traverse all page images, the target information contained in the complex document to be extracted is identified and compiled.

[0016] By inputting two consecutive page images into a multimodal large model as page pairs, the system can quickly locate page regions containing potential target information and generate a set of associated pages.

[0017] In one embodiment, constructing a cross-page dependency graph based on the set of associated pages includes:

[0018] The images of related pages are parsed using a graph traversal algorithm;

[0019] For each reference relationship from the source node to the target node, create a directed edge, and build a directed graph based on all directed edges as a cross-page dependency graph.

[0020] In a preferred embodiment, the graph traversal algorithm includes a breadth-first search algorithm.

[0021] In one embodiment, for the extraction task to be executed, a multimodal large model is used again to extract deep information from the dependency graph related to the extraction task, including:

[0022] S41. Query the dependency graph related to the extraction task to be executed, and stitch together the image files of all pages related to the extraction task to be executed into a whole visual input file;

[0023] S42. Submit the overall visual input file and the preset extraction prompts together to the multimodal large model for depth information extraction.

[0024] In one embodiment, the method further includes:

[0025] When the total number of pages that an extraction task depends on exceeds a preset threshold, the extraction task to be executed is decomposed.

[0026] In one embodiment, the method further includes:

[0027] The prompt words were optimized and enhanced.

[0028] Secondly, this application also proposes a complex document intelligent extraction system based on a multimodal large model, the system comprising:

[0029] The document preprocessing module is configured to convert the complex document to be extracted into a set of page images with predetermined requirements;

[0030] The quick filtering module is configured to perform preliminary filtering on the set of page images based on a multimodal large model to obtain a set of related pages;

[0031] The dependency graph construction module is configured to construct a cross-page dependency graph based on the set of associated pages;

[0032] The deep extraction module is configured to extract deep information from the dependency graph related to the extraction task to be executed by using a multimodal large model again.

[0033] Thirdly, this application further proposes an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the intelligent document extraction method based on a multimodal large model as described in any of the preceding claims.

[0034] Fourthly, this application finally proposes a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the intelligent document extraction method based on a multimodal large model as described in any of the preceding claims.

[0035] (III) Beneficial Effects

[0036] This invention provides a method, system, device, and storage medium for intelligent extraction of complex documents based on a multimodal large model. Compared with existing technologies, it has the following advantages:

[0037] A method for intelligent extraction of complex documents based on a multimodal large model includes: converting the complex document to be extracted into a set of page images with predetermined requirements; performing preliminary screening of the page image set based on the multimodal large model to obtain a set of related pages; constructing a cross-page dependency graph based on the set of related pages; and, for the extraction task to be executed, further extracting deep information from the dependency graph related to the extraction task using the multimodal large model. This method utilizes a multimodal large model and a two-stage extraction strategy of preliminary rapid screening and deep information extraction to improve the efficiency and accuracy of information extraction; through the cross-page dependency graph construction mechanism, it solves the problem of information fragmentation, significantly improving the intelligence and accuracy of complex document extraction. Attached Figure Description

[0038] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0039] Figure 1 This is an embodiment of an intelligent document extraction method based on a multimodal large model according to this application;

[0040] Figure 2 This is a flowchart of the intelligent document extraction method based on a multimodal large model in the embodiments of this application;

[0041] Figure 3 This is a schematic diagram of dependency graph construction in an embodiment of this application;

[0042] Figure 4 This is a schematic diagram of context-aware overall information extraction in the embodiments of this application;

[0043] Figure 5 This is a schematic diagram illustrating the self-optimization of prompt words in the embodiments of this application;

[0044] Figure 6 This is a schematic diagram of the structured information extraction results output in the embodiments of this application;

[0045] Figure 7 A bar chart comparing the performance indicators of the method in this application and existing information extraction technologies. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention are described clearly and completely. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0047] In modern medical practice, clinical practice guidelines, such as those published by the National Comprehensive Cancer Network (NCCN), are the recognized clinical decision-making standards in oncology and other medical fields. These guidelines are densely packed with information, frequently updated, and contain critical treatment information directly related to patient lives. However, a core challenge is that up to 80% of this critical medical information exists in unstructured or semi-structured free text formats. This forces clinicians to spend a significant amount of time manually searching and retrieving information, a process that is not only inefficient but also prone to errors, severely hindering the timely and accurate application of the latest and best clinical practices.

[0048] To address this issue, the industry has developed various automated information extraction technologies, but all of them have significant limitations.

[0049] First, traditional Optical Character Recognition (OCR) technology exhibits fundamental shortcomings when processing complex medical documents. OCR is inherently a "fragile" technology, often failing when faced with the complexity of real-world medical documents. Its deficiencies include at least the following: 1) Poor handling of layout complexity: Medical guidelines commonly contain complex tables, multi-column layouts, flowcharts, and embedded charts. Traditional OCR systems struggle to correctly interpret these complex visual structures, frequently outputting chaotic, disordered, or even meaningless text streams, leading to the loss or misalignment of crucial information. 2) Lack of contextual understanding: OCR technology remains at the character recognition level, completely lacking semantic or contextual understanding capabilities. It cannot distinguish whether a string is a drug name or a patient's name, nor can it understand the relationship between the value of a cell in a table and the header of another column. 3) Heavy dependence on image quality: In actual clinical workflows, poor document scanning quality, diverse fonts, and low text contrast are very common. Under these conditions, OCR performance deteriorates catastrophically, severely impacting the accuracy of the extracted results. 4) Inherent Information Loss: The process of converting an information-rich visual page into a linear text stream is inherently a lossy operation. Important contextual information conveyed by non-textual elements, such as page layout, spatial relationships between elements, charts and graphs, and text formatting (e.g., bold, italics), is completely discarded. This "text-first" processing fundamentally loses the complete information of the document.

[0050] Secondly, even if perfect text could be obtained through some means, subsequent reliance solely on Natural Language Processing (NLP) technologies would face significant challenges: 1) Domain-specific semantic complexity: Medical language is filled with technical terms, abbreviations, and highly subtle expressions, making it difficult for general NLP models to accurately understand and process. 2) Information fragmentation and cross-page referencing challenges: This is the most intractable and unresolved problem in existing technologies. In medical guidelines, complete, actionable information (e.g., all applicable conditions for a chemotherapy regimen, including pathological type, tumor stage, treatment line, genotype, and previous treatment history) is often fragmented and scattered across multiple pages of the document. This fragmented information is interconnected through explicit textual references or implicit symbols (such as footnotes). Existing NLP systems lack a stable and reliable mechanism to automatically identify, track, and integrate this scattered information, thus failing to form a coherent and complete unit of knowledge. This directly leads to incomplete information extraction and may even cause serious medical risks due to the omission of key constraints.

[0051] In recent years, the emergence of large language models (LLMs) and multimodal large language models (MLLMs) has brought significant progress to the field, demonstrating their great potential in understanding medical text and images. However, current cutting-edge applications, such as some Retrieval-Augmented Generation (RAG) systems developed for NCCN guidelines, still suffer from fundamental flaws. These systems typically rely on a preprocessing step, converting the original PDF document into text blocks or JSON objects. This means their AI inference process is still based on the lossy textual abstraction rather than the original visual form of the document, thus inheriting the fundamental shortcomings of traditional OCR workflows. Furthermore, it is well known that LLMs significantly degrade in performance when faced with excessively long or complex contextual inputs (i.e., the "lost in the middle" problem), posing a major obstacle to processing complex documents with numerous cross-page dependencies.

[0052] Based on this, the embodiments of this application provide a method, system, device and storage medium for intelligent extraction of complex documents based on a multimodal large model, which at least solves the problems of information fragmentation and difficulty in cross-page referencing, and inaccurate information extraction, and achieves the goal of efficient, accurate and self-optimizing intelligent information extraction.

[0053] The technical solution in this application is to solve the above-mentioned technical problems, and the general idea is as follows:

[0054] To address the aforementioned problems in existing technologies, this invention provides an intelligent document extraction method based on a multimodal large model, achieving its goal through an innovative combination of technologies. This combination includes: a two-stage extraction strategy to balance efficiency and depth; a cross-page dependency graph construction mechanism to address information fragmentation; a context-aware holistic information extraction method to ensure the model reasons within a complete visual context; a dynamic task decomposition and dimensionality reduction technique to address the performance challenges posed by complex tasks; and a self-optimizing closed loop based on human feedback prompts to enable continuous learning and accuracy improvement. This invention overcomes the limitations of existing technologies, providing an efficient, accurate, and self-optimizing information extraction solution.

[0055] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific implementation methods.

[0056] It should be noted that the intelligent extraction method, system, device, and storage medium for complex documents based on a multimodal large model proposed in this application are applicable to the intelligent extraction of required content from complex documents, including but not limited to the intelligent extraction of drug regimens from complex medical documents (such as clinical practice guidelines), and also include the extraction of required information from guidelines and other complex documents in other fields. To facilitate the explanation of the technical solution of this application, the following embodiments use the intelligent extraction of drug regimens from clinical practice guidelines as an example to illustrate the implementation process of the technical solution of this application in detail. However, this example of a specific application scenario is not intended to limit the scope of protection of this application.

[0057] Example 1:

[0058] Firstly, this invention proposes an intelligent document extraction method based on a multimodal large model, see [link to relevant documentation]. Figure 1 , Figure 2 The method includes:

[0059] S1. Convert the complex document to be extracted into a set of page images according to the predetermined requirements;

[0060] S2. Based on the multimodal large model, perform preliminary screening on the set of page images to obtain the set of associated pages;

[0061] S3. Construct a cross-page dependency graph based on the aforementioned set of related pages;

[0062] S4. For the extraction task to be executed, the multimodal large model is used again to extract deep information from the dependency graph related to the extraction task to be executed.

[0063] The following is in conjunction with the appendix Figure 1-7 The following details the implementation process of an embodiment of the present invention, including explanations of the specific steps S1-S4.

[0064] S1. Convert the complex document to be extracted into a set of page images according to the predetermined requirements.

[0065] Receive a medical document (e.g., a PDF file of an NCCN guideline) and preprocess it. The preprocessing process includes:

[0066] The input document is converted into a unified, serialized, high-resolution set of page images. For example, the set of page images can be numbered P1, P2, ..., Pn according to the page numbering order of the medical document, serving as the standard input format for all subsequent processing steps. The target medical document can be in various common formats, such as PDF and DOCX.

[0067] S2. Based on the multimodal large model, perform preliminary screening on the set of page images to obtain the set of associated pages.

[0068] This step is the first stage of the two-stage extraction strategy, used to extract global metadata and perform a fast, coarse-grained, and low-cost visual scan of all page image sets obtained in step S1. In specific implementation:

[0069] First, by presenting each page of the image set to a multimodal large model (MLLMs), a list containing all drug names in the document is constructed. Specifically, the MLLMs iterate through all page images to identify and compile a list of all drug names appearing in the medical document as metadata.

[0070] Next, using page pairs (Pi, Pi+1) as units, queries are posed to MLLMs to quickly identify pages that may contain drug regimen descriptions or tables. All these pages constitute the set of associated pages. A page pair can be understood as a document in image format. The document can be Word, PDF, or other formats, which are then converted into image formats. Word documents can be saved as PDFs using Windows, and others can be printed as PDFs. PDFs can be converted to JPG image format using the open-source Python package pdfplumber, with each page converted into one image. A page pair consists of two or more consecutive image files.

[0071] Specifically, two consecutive page images (i.e., page pairs) are input into MLLMs as a unit, and a classification question is posed, such as: "Does the image contain drug regimen information? Please answer 'yes' or 'no'." This allows for the rapid location of page regions containing potential target information, and all page regions containing potential target information form an associated page set.

[0072] It's important to note that the first stage of processing mimics the workflow of human experts: when faced with a guide hundreds of pages long, experts don't read it linearly from beginning to end, but first perform rapid "triage": using the table of contents, indexes, chapter titles, and visual structures (such as tables and charts) to quickly locate potentially relevant pages. This computational simulation of the efficient "triage" strategy can quickly filter out a large amount of irrelevant content, avoiding wasting valuable computational resources on invalid information. This biomimetic design makes the entire system inherently more efficient and scalable than traditional brute-force methods.

[0073] S3. Construct a cross-page dependency graph based on the set of associated pages.

[0074] This step is used to automate the discovery and modeling of cross-page dependencies, thereby constructing a cross-page dependency graph, such as... Figure 3 As shown.

[0075] An advanced graph traversal algorithm is employed to perform deep analysis on the set of related pages selected in step S2 (i.e., the first stage). Preferably, breadth-first search (BFS) is used to analyze the related page images. Specifically, the visual and textual understanding capabilities of MLLMs are utilized to identify explicit and implicit references on the pages. Explicit references refer to elements on the page such as "see..." Figure 2 Text strings such as "See page 112 for details" and "Refer to Appendix B"; implicit references refer to superscript symbols appearing on the page, such as footnotes or endnotes (e.g., aa, bb, ...). These symbols (etc.) may point to comments on the same page or to a separate comment area at the end of the document.

[0076] For each reference relationship from a source node (such as a sentence on page Px) to a target node (such as page Py), a directed edge is created, ultimately constructing a complete directed graph that represents the flow of document information.

[0077] Reference Figure 3 This diagram illustrates the dependency graph construction in this embodiment. Pages marked as relevant in the first stage undergo deep parsing. Leveraging the powerful visual and textual joint understanding capabilities of MLLMs, all cross-page reference clues are located and parsed on the page image. This includes explicit textual instructions and implicit visual symbols (such as footnote superscripts). Whenever a dependency relationship from page Px to page Py is found, a directed edge is added to the graph data structure. Through graph traversal algorithms such as breadth-first search (BFS), the system can discover all transitive dependency chains, for example, page A references page B, and page B references page C, thus constructing a complete and accurate directed graph reflecting the knowledge relationships within the document. This completes the construction of the cross-page dependency graph.

[0078] S4. For the extraction task to be executed, the multimodal large model is used again to extract deep information from the dependency graph related to the extraction task to be executed.

[0079] This step is the second stage of the two-stage extraction strategy, used to orchestrate and execute a refined information extraction process. It is used when a specific extraction task needs to be performed, such as extracting all applicable conditions for "Solution X". Figure 4 As shown, the specific steps include:

[0080] S41. Query the dependency graph related to the extraction task to be executed, and stitch together the image files of all pages related to the extraction task to be executed into a whole visual input file.

[0081] First, query the dependency graph constructed in step S3 to find the main page that defines "Solution X" and the set of all pages that it depends on.

[0082] Next, the image files for all pages in this collection are retrieved and stitched together into a logically coherent visual input. This input can be a very large composite image or an ordered sequence of images.

[0083] S42. Submit the overall visual input file and the preset extraction prompts together to the multimodal large model for depth information extraction.

[0084] The aforementioned composite visual input, along with a highly targeted prompt, is submitted to MLLMs for deep extraction. Preferably, the prompt can be in the form of: "Based on the provided set of page images, please extract all applicable conditions for the 'cisplatin + pemetrexed' regimen, including but not limited to biomarkers, cancer stage, performance status requirements, and any contraindications mentioned in footnotes or notes."

[0085] It should be noted that the "visual-first" method proposed in this embodiment completely abandons the information-loss-inducing preprocessing step of reducing visual documents to linear text in traditional methods. By directly allowing the model to "see" the original page layout that is no different from what human experts see, the model can simultaneously use the text content and the rich visual context conveyed by layout, spatial relationships, symbols, etc. to make inferences, thereby greatly improving the accuracy and completeness of extraction.

[0086] In one embodiment, to alleviate the model's "cognitive burden" and ensure model performance, the system performs dynamic task decomposition for complex extraction tasks. In this case, the method further includes:

[0087] S5. When the total number of pages that the extraction task to be executed depends on exceeds a preset threshold, the extraction task to be executed is decomposed.

[0088] Specifically, during step S4, a complexity check is performed on the extraction task to be executed. If the total number of pages a single extraction task depends on exceeds a preset threshold (e.g., more than 10 pages), the task decomposition mechanism will intervene. For example:

[0089] When decomposing tasks, a large, complex task (e.g., "Extract all information about solution X from these 20 pages") is intelligently broken down into a series of smaller, more focused semantic subtasks. For example:

[0090] Subtask 1: Input the three most relevant page images of the biomarker and ask: "Extract only the biomarker applicable conditions for scheme X".

[0091] Subtask 2: Input the two page images most relevant to usage and dosage, and ask: "Extract only the dosage and dosing regimen for regimen X".

[0092] It's important to note that the task decomposition process described above is not simply text chunking, but rather a semantically and task-driven contextual dimensionality reduction. It provides each subquery with input boasting a very high signal-to-noise ratio, allowing the model to focus all its reasoning capabilities on solving a clear and well-defined problem, thus significantly improving extraction accuracy and robustness in complex scenarios. Furthermore, the complexity checking and task decomposition mechanisms avoid providing excessive and cluttered information to the large language model (LLM), which could dilute key signals, lead to model "attention" distraction and performance degradation, and prevent cognitive overload.

[0093] In one embodiment, to enable the system to learn and optimize itself, the method further includes:

[0094] S6. Optimize and enhance the prompt words.

[0095] The structured results extracted by the system are submitted to medical experts for review. When an expert finds an error (e.g., missing an applicable condition hidden in a page footnote), the expert can submit a correction record through the feedback receiving interface. This record includes an error description and cause analysis, such as: Error type: Information omission. Details: Failed to extract an applicable condition for "Solution X". Source: Footnote "cc" on page 25. Expert analysis: The model appears to ignore the footnote at the bottom of the page indicated by the superscript.

[0096] Based on expert feedback, a meta-cue is constructed, and an LLM (Limited Language Management) is invoked for inference: "The current extracted cue word is '[Original cue word text]'. Using this cue word resulted in a 'model ignores superscript footnotes' error. Please generate an improved cue word based on expert analysis to resolve this issue." The LLM will return an enhanced cue word, for example, by adding an explicit instruction to the original cue word: "Pay special attention to superscript footnote symbols in the text (such as 'aa', 'bb', '...')." (The text then abruptly shifts to a different topic, mentioning a footnote or endnote, and integrating it into the final applicable conditions.) This newly generated, improved prompt will be reviewed and stored in the prompt library for use in subsequent tasks. The prompt self-optimization process is detailed as follows: Figure 5 As shown.

[0097] It's important to note that this closed-loop feedback mechanism transforms the system from a static tool into a dynamic, learning "living entity." In reality, the content and format of medical guidelines are not static but frequently updated. Traditional fine-tuning models require expensive and time-consuming retraining to adapt to these changes. This embodiment, however, offers an extremely agile adaptation method. When a new version of the guidelines introduces a new annotation standard, it can learn this new rule through a single expert correction, without any modification to the underlying model. More importantly, this process captures and solidifies the experts' tacit knowledge—the subtle rules and patterns they use when interpreting complex documents. This means that the method in this embodiment not only learns the literal content of the guidelines but also learns how to read them like an expert. This creates a virtuous cycle: the more intelligent the system, the better the initial extraction results, allowing experts to focus on more subtle errors, and this high-quality feedback further enhances the system's intelligence. The system co-evolves with the content it processes and its users, thereby meeting the stringent requirements of domain specificity and reliability in medical information processing.

[0098] This completes the entire process of the intelligent document extraction method based on a multimodal large model proposed in this embodiment and its preferred embodiment.

[0099] The following section explains the intelligent document extraction method based on a multimodal large model proposed in this embodiment and its preferred embodiment by describing the specific workflow of an information extraction task that may be encountered in a practical application scenario.

[0100] Information extraction task: Extract complete treatment regimen information for "Doxorubicin + Cyclophosphamide followed by Paclitaxel" (AC-T regimen) from an NCCN Breast Cancer Clinical Practice Guideline PDF document.

[0101] Step 1: Convert the complex document into a set of page images according to the predetermined requirements, and filter out the set of pages related to the AC-T scheme description.

[0102] The system receives the NCCN Breast Cancer Clinical Practice Guidelines PDF document, preprocesses the PDF document, and converts it into a series of page images.

[0103] The system iterated through all pages, constructing a list of drug names containing keywords such as "Doxorubicin," "Cyclophosphamide," and "Paclitaxel." Subsequently, during a scan of page pairs, the system identified a table on page BC-25 that was highly likely to contain a detailed description of the AC-T regimen, and therefore marked that page as highly relevant.

[0104] Step 2: Perform deep analysis on the images on page BC-25 to construct a dependency graph.

[0105] A deep analysis of the image on page BC-25 revealed, for example, a superscript symbol "cc" next to the AC-T protocol description. Simultaneously, in a line of text below the table, it identified the text "Applicable to HER2-negative disease (see page BC-10)". Based on these findings, two directed edges were created: one from page BC-25 to page BC-10, and the other from page BC-25 to the page containing the footnote defining "cc" (let's assume page BC-90). In the footnote area of ​​page BC-90, the system found the corresponding text for "cc": "Applicable only to patients with normal cardiac function".

[0106] Step 3: Extract the pages related to the AC-T scheme and integrate them into a unified visual input (composite image or ordered image sequence).

[0107] When the system receives the task of extracting the applicable conditions of the AC-T scheme, it queries the dependency graph and determines that it needs to integrate the information from the three pages BC-25, BC-10, and BC-90. Then, it obtains the images from these three pages and combines them into a unified visual input.

[0108] Step 4: Determine whether the total number of pages required for the extraction task exceeds a preset threshold, and decide whether to perform task decomposition.

[0109] Determine whether the total number of pages required for this extraction task exceeds a preset threshold. Since the number of required pages is 3, which does not exceed the preset threshold, the task decomposition operation is not performed.

[0110] Step 5: Output structured knowledge related to the extraction task to be performed.

[0111] Deep extraction is performed. The MLLMs model receives a composite input consisting of three pages of images and extraction cues, and infers within a complete visual context. Because inference is performed within a complete visual context, the system sees not only the table content on page BC-25, but also the definition of HER2 states on page BC-10, and the footnotes about cardiac function on page BC-90. Ultimately, it generates... Figure 6 The structured JSON object shown.

[0112] Through this information extraction task, the detailed example above clearly demonstrates how this embodiment, through its unique combination of technologies, successfully and accurately extracts structured knowledge from information fragments scattered across different pages and existing in different forms (tables, body text, footnotes). This is significantly superior to existing technologies that may miss footnote information or fail to effectively integrate cross-page information.

[0113] To systematically and quantitatively evaluate the performance of the method in this embodiment in complex document information extraction tasks, the following specific experiments demonstrate its effectiveness.

[0114] This experiment aims to compare the method described in the above embodiments with two mainstream existing technologies (traditional OCR+NLP and text RAG based on pre-parsed data (such as JSON)). The evaluation will focus on a series of standard evaluation metrics in the field of information extraction, including accuracy, precision, recall, and F1 score, to comprehensively verify the superiority of the method in terms of extraction accuracy and completeness. Specific comparative experimental data can be found in Table 1 and... Figure 7 Table 1 and Figure 7 This clearly demonstrates how the method proposed in this embodiment systematically solves many pain points of existing technologies, thereby achieving success in complex medical document information extraction tasks. Specific experimental details are as follows:

[0115] Experimental dataset: Fifteen different versions of the National Comprehensive Cancer Network (NCCN) clinical practice guidelines were selected as the test corpus.

[0116] Experimental Task: 100 predefined information extraction tasks were performed from the dataset. Each task required extracting complete treatment plan information, including its applicable conditions, constituent drugs, dosage, contraindications, etc. All tasks were cross-labeled by three oncology experts to generate a highly reliable "Gold Standard". A positive example was defined as successfully extracting a complete and correct information item present in a "Gold Standard" from the document.

[0117] Comparison of technical solutions:

[0118] Solution A (Traditional OCR+NLP): Use an industry-leading OCR engine to convert PDF documents into plain text, and then apply Named Entity Recognition (NER) and Relation Extraction (RE) models based on Transformer architecture (such as BERT) to extract information.

[0119] Option B (Text RAG Based on Pre-parsed Data): First, the PDF document is parsed into JSON or Markdown format with basic layout information using a tool. Then, the Retrieval Enhanced Generation (RAG) large language model is used to extract information from the parsed text blocks through a question-and-answer approach.

[0120] Solution C (the method of this embodiment): The complete technical process described in this embodiment is adopted, that is, the operation is performed directly on the page image, and the task is completed through a two-stage extraction strategy, construction of a cross-page dependency graph, and context-aware overall information extraction.

[0121] Evaluation index definition:

[0122] Precision = TP / (TP + FP), meaning the proportion of correct information extracted by the model. High precision indicates that the model "doesn't give wrong information".

[0123] Recall (TP / FN) means that the model successfully extracts the proportion of information that should be extracted from the standard answer. A high recall rate indicates that the model "does not miss anything".

[0124] F1 score (F1 - Score) = 2 (Precision Recall / (Precision + Recall), meaning: the harmonic mean of precision and recall, is the most commonly used metric for measuring the overall performance of a model.

[0125] Accuracy = (TP + TN) / (TP + FP + TN + FN), meaning: the proportion of correctly predicted samples (including correctly extracted and correctly unextracted samples) out of the total sample (Note: TP = true positives, FP = false positives, FN = false negatives, TN = true negatives).

[0126] Table 1: Comparison of Information Extraction Methods

[0127]

[0128] From Table 1 and Figure 7 As can be seen, the intelligent extraction method for complex documents based on a multimodal large model proposed in this embodiment is superior to existing related technologies in all aspects, including complex layout processing, context understanding, cross-page dependency issues, adaptability to guideline updates, and dependence on manual annotation. The accuracy of information extraction is also significantly higher than that of existing technologies.

[0129] Example 2:

[0130] Secondly, the present invention also provides a complex document intelligent extraction system based on a multimodal large model, the system comprising:

[0131] The document preprocessing module is configured to convert the complex document to be extracted into a set of page images with predetermined requirements;

[0132] The quick filtering module is configured to perform preliminary filtering on the set of page images based on a multimodal large model to obtain a set of related pages;

[0133] The dependency graph construction module is configured to construct a cross-page dependency graph based on the set of associated pages;

[0134] The deep extraction module is configured to extract deep information from the dependency graph related to the extraction task to be executed by using a multimodal large model again.

[0135] The document preprocessing module, fast filtering module, dependency graph construction module, and deep extraction module described above perform the corresponding steps and processes according to the method of Embodiment 1 and its preferred embodiments.

[0136] Specifically:

[0137] In one embodiment, the deep extraction module also includes a context composition module. This module dynamically constructs the model's input based on a specific extraction task and dependency graph. When specific information (such as the applicability conditions of "drug regimen X") needs to be extracted, this module queries the dependency graph to obtain the main page defining that information and all pages that it directly or indirectly depends on. Then, it retrieves the image files (e.g., Px, Py, Pz, ...) of these pages and combines them into a single, unified visual canvas or an ordered sequence of images as input to MLLMs.

[0138] In one embodiment, the system further includes a task decomposition module. This module aims to address the performance degradation caused by excessive page dependencies. It incorporates a complexity evaluation logic; for example, the module is activated when the number of pages a task depends on exceeds a preset threshold (e.g., 10 pages). Instead of feeding all pages to the model at once, it decomposes the original complex query into multiple semantically independent subtasks. For example, "extract all information about scheme X" is decomposed into "extract only the biomarker conditions of scheme X," "extract only the usage and dosage of scheme X," etc. Simultaneously, it matches a minimal subset of relevant pages as input for each subtask.

[0139] In one embodiment, the system further includes a prompt word management and optimization module: this module is key to realizing the system's self-learning capability. Correspondingly, the system also includes:

[0140] The prompt word library stores a carefully designed set of prompt word templates for performing different types of extraction tasks. This library supports version control to track and manage the evolution of the prompt words.

[0141] Feedback Receiving Interface: Provides a user interface that allows human review experts (such as clinicians) to submit feedback on extraction errors. Feedback includes not only the erroneous results but also a natural language description of the reasons for the errors.

[0142] The prompt word optimizer submodule receives expert feedback and utilizes a large language model (LLM) to perform meta-level inference. It submits a meta-prompt to the LLM, similar to: "Analyze the following error: The user feedback model failed to extract the footnote information pointed to by the superscript 'aa' when processing page X. Based on this analysis, generate an improved extraction prompt word to ensure correct handling of similar situations in the future." The LLM then generates an optimized prompt word suggestion based on this.

[0143] Structured Output Module: This module is responsible for formatting the potentially unstructured text results returned by the deep extraction submodule, ultimately generating a standard, structured data object (such as a JSON file). This object clearly contains all the elements of the drug regimen, such as the regimen name, constituent drugs, applicable conditions, and level of evidence, and includes source information (such as the original page number) to facilitate traceability and verification.

[0144] It is understood that the intelligent extraction system for complex documents based on multimodal large models provided in this embodiment of the invention corresponds to the intelligent extraction method for complex documents based on multimodal large models described above. The explanations, examples, and beneficial effects of the relevant content can be referred to the corresponding content in the intelligent extraction method for complex documents based on multimodal large models, and will not be repeated here.

[0145] Example 3:

[0146] Thirdly, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps of the intelligent document extraction method based on a multimodal large model as described in any of the above embodiments and their preferred embodiments. The method mainly includes:

[0147] S1. Convert the complex document to be extracted into a set of page images according to the predetermined requirements;

[0148] S2. Based on the multimodal large model, perform preliminary screening on the set of page images to obtain the set of associated pages;

[0149] S3. Construct a cross-page dependency graph based on the aforementioned set of related pages;

[0150] S4. For the extraction task to be executed, the multimodal large model is used again to extract deep information from the dependency graph related to the extraction task to be executed.

[0151] It is understood that the electronic device for intelligent extraction of complex documents based on a multimodal large model provided in this embodiment of the invention corresponds to the above-mentioned method and system for intelligent extraction of complex documents based on a multimodal large model. The explanation, examples, and beneficial effects of the relevant content can be referred to the corresponding content in the method and system for intelligent extraction of complex documents based on a multimodal large model, and will not be repeated here.

[0152] Example 4:

[0153] Fourthly, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the intelligent document extraction method based on a multimodal large model as described in any of the above embodiments and their preferred embodiments, the method comprising:

[0154] S1. Convert the complex document to be extracted into a set of page images according to the predetermined requirements;

[0155] S2. Based on the multimodal large model, perform preliminary screening on the set of page images to obtain the set of associated pages;

[0156] S3. Construct a cross-page dependency graph based on the aforementioned set of related pages;

[0157] S4. For the extraction task to be executed, the multimodal large model is used again to extract deep information from the dependency graph related to the extraction task to be executed.

[0158] It is understood that the intelligent extraction and storage medium for complex documents based on multimodal large models provided in this embodiment of the invention corresponds to the aforementioned intelligent extraction method and system for complex documents based on multimodal large models. The explanations, examples, and beneficial effects of the relevant content can be referred to the corresponding content in the intelligent extraction method and system for complex documents based on multimodal large models, and will not be repeated here.

[0159] In summary, compared with existing technologies, it has the following beneficial effects:

[0160] 1. The complex document intelligent extraction technology based on a multimodal large model proposed in this application converts the complex document to be extracted into a set of page images with predetermined requirements; performs preliminary screening of the page image set based on the multimodal large model to obtain a set of related pages; constructs a cross-page dependency graph based on the set of related pages; and, for the extraction task to be executed, further extracts deep information from the dependency graph related to the extraction task using the multimodal large model. The technology of this application, utilizing a multimodal large model and a two-stage extraction strategy of preliminary rapid screening and deep information extraction, improves the efficiency and accuracy of information extraction; and solves the problem of information fragmentation through the cross-page dependency graph construction mechanism, significantly improving the intelligence and accuracy of complex document extraction.

[0161] 2. The intelligent document extraction technology based on a multimodal large model proposed in this application improves efficiency and accuracy through a two-stage strategy: First, in the first stage, candidate pages are screened and key entities are initially identified through rapid, coarse-grained visual scanning. Then, computationally intensive, fine-grained information extraction is performed only on a subset of highly relevant pages. This design optimizes the use of computing resources while improving the overall extraction accuracy.

[0162] 3. The intelligent document extraction technology based on multimodal large model proposed in this application solves the problem of information fragmentation through a cross-page dependency graph construction mechanism: by analyzing textual and non-textual cues in the document (such as citation text, footnotes, etc.), a complete graph representing how information is related and dependent on different pages is automatically generated, thereby fundamentally solving the problem of scattered information distribution.

[0163] 4. The complex document intelligent extraction technology based on a multimodal large model proposed in this application ensures that the model performs reasoning within a complete visual context by setting a context-aware overall information extraction method: combining the images of all relevant pages on which an extraction task depends into a single, composite visual input, and providing it as a whole to the multimodal large model. This ensures that the model can "see" the complete and lossless context when extracting any specific information point.

[0164] 5. The intelligent document extraction technology based on a multimodal large model proposed in this application sets up a dynamic task decomposition and dimensionality reduction mechanism, which can easily cope with the performance challenges brought by complex tasks: a complex extraction task involving many dependent pages is automatically decomposed into multiple smaller, more manageable semantic subtasks, and each subtask is provided with a minimized context input. This effectively alleviates the performance degradation problem of large models when processing extremely long contexts, ensuring the effectiveness of the system when facing highly complex documents.

[0165] 6. The intelligent document extraction technology based on a multimodal large model proposed in this application sets up a self-optimizing closed loop of prompt words based on human feedback to achieve continuous learning and accuracy improvement of the system: review feedback from medical experts (such as reports on extraction errors) is used to automatically analyze the reasons for failure and generate optimized and more accurate prompt words. This enables the system to continuously learn and adapt with use, achieving continuous iterative improvement in performance.

[0166] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0167] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for intelligent extraction of complex documents based on a multimodal large model, characterized in that, The method includes: The complex document to be extracted is converted into a set of page images according to the predetermined requirements; The set of page images is initially filtered based on a multimodal large model to obtain a set of related pages; Construct a cross-page dependency graph based on the aforementioned set of related pages; For the extraction task to be executed, the multimodal large model is used again to extract deep information from the dependency graph related to the extraction task to be executed. The cross-page dependency graph is a directed graph reflecting the cross-page dependency relationships between related pages; the construction of the cross-page dependency graph based on the set of related pages includes: The images of related pages are parsed using a graph traversal algorithm; For each reference relationship from the source node to the target node, create a directed edge, and build a directed graph based on all directed edges as a cross-page dependency graph.

2. The method as described in claim 1, characterized in that, The page image set is initially filtered based on a multimodal large model to obtain a set of associated pages, including: By using a multimodal large model to traverse all page images, the target information contained in the complex document to be extracted is identified and compiled. By inputting two consecutive page images into a multimodal large model as page pairs, the model can quickly locate page regions containing potential target information and generate a set of associated pages.

3. The method as described in claim 1, characterized in that, The graph traversal algorithm includes the breadth-first search algorithm.

4. The method as described in claim 1, characterized in that, For the extraction task to be executed, the multimodal large model is used again to extract deep information from the dependency graph related to the extraction task, including: S41. Query the dependency graph related to the extraction task to be executed, and stitch together the image files of all pages related to the extraction task to be executed into a whole visual input file; S42. Submit the overall visual input file and the preset extraction prompts together to the multimodal large model for depth information extraction.

5. The method according to any one of claims 1-4, characterized in that, The method further includes: When the total number of pages that an extraction task depends on exceeds a preset threshold, the extraction task to be executed is decomposed.

6. The method as described in claim 4, characterized in that, The method further includes: The prompt words were optimized and enhanced.

7. A complex document intelligent extraction system based on a multimodal large model, characterized in that, The system includes: The document preprocessing module is configured to convert the complex document to be extracted into a set of page images with predetermined requirements; The quick filtering module is configured to perform preliminary filtering on the set of page images based on a multimodal large model to obtain a set of related pages; The dependency graph construction module is configured to construct a cross-page dependency graph based on the set of associated pages; the cross-page dependency graph is a directed graph reflecting the cross-page dependency relationships between associated pages; the construction of the cross-page dependency graph based on the set of associated pages includes: parsing the images of associated pages using a graph traversal algorithm; creating a directed edge for each reference relationship from a source node to a target node, and constructing a directed graph as the cross-page dependency graph based on all directed edges; The deep extraction module is configured to extract deep information from the dependency graph related to the extraction task to be executed by using a multimodal large model again.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the intelligent document extraction method based on a multimodal large model as described in any one of claims 1-6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the intelligent document extraction method based on a multimodal large model as described in any one of claims 1-6.

Images