Multi-modal fusion universal document intelligent credible extraction method and system

Abstract

The invention provides a multi-modal fusion universal document intelligent credible extraction method and system. The method comprises the following steps: firstly, obtaining an original document; analyzing scene information of the original document, and determining a target sub-model corresponding to the scene information; segmenting the original document by adopting an overlapping segmentation merging algorithm to obtain fragmented files; wherein segmentation parameters of the overlapping segmentation merging algorithm are adjusted according to document features and scene features; and finally, obtaining a document extraction result according to the fragmented files and the large model of the target sub-model of the universal document intelligent credible extraction method with multi-modal fusion. According to the method, through scene recognition, adaptive document and scene feature overlapping segmentation, long document information integrity guarantee of multi-scene document extraction is realized, scene adaptation flexibility is improved, and extraction result accuracy and verifiability are enhanced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular to a multi-modal fusion general document intelligent and reliable extraction method and system. BACKGROUND

[0002] In the context of the continuous deepening of social digital transformation, the demand for key information extraction of unstructured documents in multiple fields such as finance, engineering construction, personnel, medical insurance, and market supervision is increasingly urgent and presents significant differentiation. Different departments focus on different fields of extraction for the same type of document. These needs cover multiple scenarios and multiple types of documents, and are frequently affected by factors such as business expansion, making traditional manual extraction mode face the dilemma of high labor costs and low efficiency, while customized development has the disadvantages of long cycle, high cost, and difficulty in quickly responding to demand changes.

[0003] Existing technical solutions have obvious limitations and cannot meet the comprehensive needs of multi-scenario document extraction. On the one hand, the rule extraction scheme based on fixed templates has some effect on fixed format cards and forms, but the cost of data collection and labeling training is high, it cannot adapt to changes in document format, and new document types need to be redeveloped, resulting in high maintenance costs. On the other hand, the direct extraction scheme of general large models has flexibility, but is limited by the length of the context, and multiple-page documents are prone to information loss due to truncation, and the lack of location information due to pure text input makes it impossible to achieve visual verification of the extraction results, lacks a confidence verification mechanism, and the efficiency of manual review is extremely low. Both types of solutions fail to address the core issues of long document processing and information coherence, information extraction and position decoupling, fixed templates and dynamic demand mismatch. SUMMARY

[0004] The embodiments of the present application provide a multi-modal fusion general document intelligent and reliable extraction method and system, which is used to solve the problem that the existing technology fails to solve the contradiction between long document processing and information coherence.

[0005] The first aspect of the embodiments of the present application provides a multi-modal fusion general document intelligent and reliable extraction method, comprising: obtaining an original document; parsing the scene information of the original document and determining the target sub-model corresponding to the scene information; segmenting the original document using an overlapping segmentation and merging algorithm to obtain a fragmented file; wherein the segmentation parameters of the overlapping segmentation and merging algorithm are adjusted according to the document features and scene features; obtaining a document extraction result from the fragmented file and the large model with the target sub-model.

[0006] In some possible implementation manners, the original document is segmented using an overlapping segmentation and merging algorithm to obtain a fragmented file, comprising: Perform a pre-structural analysis on the original document, calculate the heading score and table score for each page, and label the page type based on the score values; Determine the basic overlap strategy based on page type; Scene features are extracted from scene information, and scene weight coefficients are determined based on scene features in order to optimize and adjust the basic overlap strategy. Extract document features from the original document based on the requirements of the current scenario. Determine the fragment length of the fragmented file based on document characteristics; Based on the adjusted base overlap strategy and fragment length, generate fragment files.

[0007] In some possible implementations, the scene information of the original document is parsed, and the target sub-model corresponding to that scene information is determined, including: Three-level feature extraction is performed on the original document to obtain basic features; the three-level feature extraction includes metadata layer extraction, text layer extraction, and visual layer extraction. Based on the basic features, calculate the scene adaptation score, and determine the scene information of the original document based on the scene adaptation score; Based on the scene information, determine the target sub-model.

[0008] In some possible implementations, document extraction results are obtained from the fragmented files and the large model with target sub-models set, including: Perform document type detection on each file segment to determine the file attributes of each file segment; Based on the file attributes of each file segment, determine the file identification strategy for each file segment; Based on the file recognition strategy, each fragment file is identified as multimodal structured data; Multimodal structured data is input into a large model with target sub-models to obtain document extraction results.

[0009] In some possible implementations, multimodal structured data is input into a large model with target sub-models to obtain document extraction results, including: Load the scenario-specific configuration and sub-model optimization resources corresponding to the current scenario to obtain a declarative configuration file; Based on the declarative configuration file, the prompt word template of the target sub-model is dynamically populated; If the type of scene information is a single industry scene, the multimodal structured data is extracted piece by piece iteratively based on the large model dominated by the target sub-model to obtain the document extraction results; If the scene information is a mixed scene, the multimodal structured data is extracted piece by piece iteratively based on the large model of multi-objective sub-model collaboration to obtain the document extraction results; Among them, the large model, which is dominated by the target sub-model, is iteratively optimized based on the target sub-model; The large model, which coordinates multiple target sub-models, distributes the collaborative tasks to each target sub-model in parallel and aggregates the extraction results of each sub-model into the large model for iterative optimization.

[0010] In some possible implementations, document type detection is performed on each fragment file to determine the file attributes of each fragment file, including: Attribute analysis is performed on the fragmented files to obtain attribute features, including document format type, text coverage, table features, and layout complexity.

[0011] Among some possible implementations, the method also includes: Based on the validation rules in the declarative configuration file, multi-level validation is performed on the extracted results. The multi-level validation includes mandatory field checks, format validation, and numerical reasonableness validation. Perform differential processing on the fields based on the validation results; Output the structured results after verification and standardization, and generate a verification report that includes the verification status and reasons for failure.

[0012] Among some possible implementations, the method also includes: Based on declarative configuration files and business logic rules in the current scenario, anti-illusion rules are parsed, including data basic validity, format and range constraints, and cross-field scenario logic consistency. Based on anti-hallucination rules and confidence assessment, multi-dimensional verification is performed on the extraction results to identify and label hallucination content. Based on the verification results, a differentiated processing strategy is implemented: confirmed hallucination content is removed, non-fictional issues are corrected, and suspected fields are marked for review. Generate a prevention and control report that includes hallucination detection results, a list of risk fields, and handling methods.

[0013] Among some possible implementations, the general intelligent and reliable document extraction method based on multimodal fusion also includes: Collect the candidate values ​​of each field, the corresponding page number list, the coordinate list, and the confidence score of the document extraction results of all fragmented files after being extracted by a large model dominated by the target sub-model or by the collaboration of multiple target sub-models. The confidence score is determined based on a comprehensive assessment of text clarity, semantic clarity, and positional rationality. Compare multiple candidate values ​​of the same field across different partitions to identify conflict types; Implement a hierarchical conflict resolution strategy based on the conflict type, and filter or combine the conflict results.

[0014] A second aspect of the present invention provides a multimodal fusion-based general document intelligent and reliable extraction system, comprising: The acquisition module is used to acquire the original document; The parsing module is used to parse the scene information of the original document and determine the target sub-model corresponding to the scene information; The segmentation module is used to segment the original document using an overlapping segmentation and merging algorithm to obtain fragmented files; the segmentation parameters of the overlapping segmentation and merging algorithm are adjusted according to document features and scene features. The extraction module is used to obtain document extraction results from the segmented files and the large model with target sub-models set.

[0015] Compared to traditional technologies, this invention provides a system that first acquires the original document; then, it parses the scene information of the original document and determines the target sub-model corresponding to the scene information; next, it uses an overlapping segmentation and merging algorithm to segment the original document, obtaining fragmented files; wherein the segmentation parameters of the overlapping segmentation and merging algorithm are adjusted according to document features and scene features; finally, based on the fragmented files and the large model with the target sub-model set, the document extraction result is obtained. This invention, through scene recognition and overlapping segmentation adapted to document and scene features, achieves enhanced integrity of long document information extraction in multi-scene document extraction, improved flexibility of scene adaptation, and enhanced accuracy and verifiability of extraction results. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating the implementation of the general intelligent and reliable document extraction method based on multimodal fusion provided in this embodiment of the invention. Figure 2 This is a schematic diagram of the structure of the multimodal fusion general document intelligent and reliable extraction system provided in the embodiments of the present invention. Detailed Implementation

[0017] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0018] Figure 1 This is a flowchart illustrating the implementation of the general intelligent and reliable document extraction method based on multimodal fusion provided in this embodiment of the invention. Figure 1 As shown, the method includes: S110, retrieve the original document; S120, parse the scene information of the original document and determine the target sub-model corresponding to the scene information; S130, The original document is segmented using an overlapping segmentation and merging algorithm to obtain fragmented files; wherein, the segmentation parameters of the overlapping segmentation and merging algorithm are adjusted according to document features and scene features; S140: Based on the segmented files and the large model with the target sub-model set, the document extraction results are obtained.

[0019] In this embodiment of the invention, acquiring the original documents is the starting point of the entire extraction process, aiming to collect various types of unstructured document data to be processed. These original documents cover multiple industry scenarios such as finance, engineering construction, human resources, medical insurance, and market supervision, specifically including various types such as procurement contracts, invoices, payment applications, construction contracts, completion acceptance reports, employee files, medical expense lists, and business licenses. The document formats include complete electronic documents with a text layer, such as editable PDFs and Word-to-PDF conversions, as well as scanned image documents, comprehensively covering document formats under different business scenarios, providing basic data support for subsequent multimodal processing and intelligent extraction.

[0020] Parsing the scene information of the original document and determining the target sub-model are key steps in achieving adaptive scene extraction. The system deeply analyzes the document scene through a three-level feature extraction mechanism: the metadata layer reads attribute information such as the document title, author, and creation software; the text layer extracts high-frequency keywords from the first three pages to construct scene keyword vectors, such as "construction permit" and "supervision," pointing to engineering construction scenarios; the visual layer analyzes page layout features and outputs scene probability distributions through a pre-trained layout classification model. Based on these features, the system utilizes a built-in scene adaptability scoring function: Score(scene) = w1 × metadata matching degree + w2 × keyword coverage + w3 × layout confidence degree.

[0021] If the score is greater than 0.8, the corresponding scene is locked and a dedicated sub-model is scheduled. If the score is less than 0.6, it falls back to the general large model base. If the score is between 0.6 and 0.8 and there are multiple candidate scenes, it is determined to be a mixed scene, and the multi-model collaborative mode is activated. The dedicated sub-model is optimized through scene-specific keyword library, dedicated prompt word templates, and spatial prior knowledge to ensure the professionalism and accuracy of domain extraction.

[0022] The algorithm employs an overlap-based segmentation and merging technique to segment the original document. Its core principle is to achieve efficient and complete segmentation of long documents through dynamic parameter adjustment. The algorithm consists of two layers: structure awareness and scene semantic optimization. The structure awareness layer first performs a pre-analysis of the document's structure, calculating the score for each page's title and table based on visual features such as text block density, font size variation, and line density. It then labels each page as a title page, table page, or regular page, determining the basic overlap strategy: 1 page overlap for regular paragraphs, 2 pages overlap for the starting page of a table, and no overlap for new chapter title pages. The scene semantic optimization layer adds scene weight coefficients, adjusting the segmentation parameters according to different scene characteristics. For example, in a financial scene, the starting page of a table overlaps by 3 pages, and regular pages overlap by 2 pages; in an engineering scene, drawing pages do not overlap, and clause pages overlap by 1 page; in a medical scene, each page is forced to overlap by 2 pages. Simultaneously, the segment length is adjusted based on the characteristics of the document in each scene: financial contracts are set to 5 pages per segment, medical records are shortened to 3 pages per segment, and non-text pages are skipped in engineering contracts. Finally, a list of segmented files is generated, balancing information integrity and processing efficiency.

[0023] The system selects an appropriate recognition strategy based on the attribute characteristics of the fragmented files, converting them into multimodal structured data. First, it detects attributes such as document format type, text coverage, presence of complex tables, and layout complexity of the fragmented files. Then, it dynamically selects the recognition mode: Editable documents with text coverage >95% use a fast mode, directly parsing built-in text objects and coordinates, achieving a processing speed 5-10 times faster than OCR mode; scanned images and documents with text coverage <80% and containing tables use a high-precision mode, simultaneously enabling OCR text recognition and layout detection models to reconstruct logical paragraphs and build table row and column relationships; simple format documents such as cards and invoices use OCR mode, calling a lightweight OCR engine to output text lines and coordinates. These three modes are deeply coupled with subsequent processing flows, ensuring a dynamic balance between extraction speed, accuracy, and resource consumption, ultimately outputting multimodal structured data containing text content, precise coordinates, semantic role labels, and structural information.

[0024] Multimodal structured data is input into a large model with target sub-models, and the final document extraction result is obtained through multi-stage processing. First, a declarative configuration file based on JSONSchema is loaded to dynamically build prompt word templates adapted to the target sub-models, clearly defining requirements such as extraction fields, data types, and validation rules. Then, extraction is iteratively performed by segment. For single-industry scenarios, a dedicated sub-model leads the extraction; in mixed scenarios, multiple sub-models process their respective domain fields in parallel, and the large model aggregates and integrates the results. During extraction, the output must be a structured result containing field values, a list of page numbers, a list of coordinates, and a confidence score. All segment results are then merged using a conflict resolution strategy that prioritizes confidence, merges across pages, and uses majority voting. Afterwards, an independent rule validation and anti-illusion engine performs multi-layered validation, including mandatory fields, format specifications, enumerated values, and numerical ranges. Invalid results are handled according to preset strategies. Finally, a post-processor performs standardization processing such as date formatting, monetary precision control, and string cleaning to generate reliable document extraction results. Meanwhile, through scenario-based annotation and linkage positioning mechanisms, a pixel-level coordinate mapping between the extracted results and the original document is established, supporting click-to-review and generating annotated documents with bookmarks, thereby improving verification and auditing efficiency.

[0025] In some embodiments, an overlapping segmentation and merging algorithm is used to segment the original document to obtain fragmented files, including: performing structural pre-analysis on the original document, calculating the title score and table score for each page, and labeling the page type according to the score values; determining a basic overlapping strategy based on the page type; extracting scene features from scene information, and determining scene weight coefficients based on the scene features to optimize and adjust the basic overlapping strategy; extracting document features from the original document according to the scene requirements under the current scene; determining the fragment length of the fragmented file based on the document features; and generating fragmented files according to the adjusted basic overlapping strategy and fragment length.

[0026] In this embodiment of the invention, an overlapping segmentation and merging algorithm is used to segment the original document. The core of this approach begins with a pre-analysis of the document's structure, which is fundamental to ensuring that the segmentation aligns with the content logic. The system analyzes the text block layout, font size variations, line density, and other visual features of the original document page by page, calculating the title score and table score for each page using a preset algorithm. The title score is determined based on dimensions such as font boldness, font size, and position distribution, while the table score is calculated based on features such as closed line areas and cell distribution. Based on the score thresholds, the system accurately marks the page as a title page, a table page, or a regular page, providing a structured basis for subsequent overlapping strategies and avoiding the drawbacks of traditional fixed segmentation that ignores the internal logic of the document.

[0027] Based on the page type identified by the tags, the system determines a targeted basic overlap strategy to achieve initial adaptation of the segmentation parameters. For ordinary pages mainly consisting of paragraph text, a 1-page overlap setting is used to ensure semantic coherence between adjacent segments while avoiding excessive redundancy. For the starting page of a table area, the overlap is increased to 2 pages to effectively prevent data breaks caused by table page breaks and ensure the integrity of table information. However, for the title page corresponding to a new chapter, no overlap is set to avoid the same title being processed repeatedly by multiple segments, reducing unnecessary computational overhead and achieving an initial balance between information integrity and processing efficiency.

[0028] Integrating scenario features is key to dynamically optimizing segmentation strategies. Adjusting scenario weight coefficients allows segmentation to better align with business needs. The system extracts core features from parsed scenario information, such as high table density in financial scenarios, numerous drawing attachments in engineering scenarios, and frequent page breaks in medical records in medical scenarios, and determines corresponding scenario weight coefficients accordingly. Based on these coefficients, the basic overlap strategy is fine-tuned: in financial scenarios, the overlap of the starting page of a table is increased to 3 pages, and the overlap of ordinary pages is adjusted to 2 pages; in engineering scenarios, overlap is disabled for drawing pages, while clause pages retain 1 page overlap; in medical scenarios, all pages are forced to have 2 pages of overlap; and in card and certificate scenarios, overlap is directly disabled, and each page is processed independently, deeply binding the segmentation strategy to scenario characteristics.

[0029] Finally, based on the business requirements of the current scenario, document features are extracted to accurately determine the segment length and generate segmented files. The system analyzes document features such as the page distribution and content type ratio of the original document, and adjusts the segment length according to the scenario's requirements for information continuity: financial contracts have an average of 30-50 pages, with a segment length of 5 pages / segment to balance token utilization and information integrity; construction contracts contain a large number of drawing pages, so non-text pages are skipped during segmentation to avoid invalid token consumption; medical records need to strictly maintain the treatment sequence, so the segment length is shortened to 3 pages / segment. Finally, the system integrates the adjusted overlap strategy and segment length to generate a list of segmented files that balances information integrity, processing efficiency, and scenario adaptability, laying a solid foundation for subsequent multimodal recognition and intelligent extraction.

[0030] In some embodiments, parsing the scene information of the original document and determining the target sub-model corresponding to the scene information includes: performing three-level feature extraction on the original document to obtain basic features; wherein, the three-level feature extraction includes metadata layer extraction, text layer extraction and visual layer extraction; calculating the scene adaptation score based on the basic features, and determining the scene information of the original document based on the scene adaptation score; and determining the target sub-model based on the scene information.

[0031] In this embodiment of the invention, the first step in parsing the scene information of the original document and determining the target sub-model is to perform three-level feature extraction on the original document: metadata layer, text layer, and visual layer, to construct a comprehensive basic feature system. Metadata layer extraction focuses on document attribute information, reading key metadata such as title, author, and creation software to provide initial clues for scene determination. Text layer extraction focuses on mining the core semantics of the document, extracting high-frequency keywords from the first three pages and constructing scene keyword vectors. Keyword weighting is used to clarify the direction of scene association; for example, keywords such as "budget number" and "treasury payment" lean towards fiscal scenarios, while keywords such as "construction permit" and "supervision" point to engineering construction scenarios. Visual layer extraction deeply analyzes page layout features, using a pre-trained layout classification model to output the probability distribution of each candidate scene, supplementing the scene determination criteria from a visual perspective. The three layers of features work together to form a complete data foundation for scene recognition.

[0032] After acquiring the three basic features, the system calculates the scene fit score using a built-in scene fit scoring function to accurately define the scene information of the original document. The scoring function is based on Score(scene) = w1 × metadata matching degree + w2 × keyword coverage + w3 × layout confidence, where w1, w2, and w3 are the weighting coefficients for metadata matching degree, keyword coverage, and layout confidence, respectively. Weighted calculations achieve a comprehensive quantitative evaluation of multi-dimensional features. Based on the calculated fit score, the system determines the scene type: if the highest-scoring scene has a Score > 0.8, the document's scene is clearly defined, and that scene is directly locked; if the Score < 0.6, the document has no obvious scene (such as a general cooperation agreement), and is classified as a vague scene; if the Score is between 0.6 and 0.8 and multiple candidate scenes exist, such as a document containing both a financial payment application and an engineering contract attachment, it is classified as a mixed scene, ensuring the accuracy and comprehensiveness of scene information determination.

[0033] Based on defined scenario information, the system dynamically matches the corresponding target sub-model to achieve precise adaptation between the scenario and the model. For a single specific scenario (Score > 0.8), the system schedules the pre-defined dedicated sub-model for that scenario as the target sub-model. The dedicated sub-model enhances the professionalism and accuracy of domain extraction by integrating scenario-specific keyword libraries, dedicated prompt word templates, and spatial prior knowledge (such as the common location of "budget number" in financial contracts). For mixed scenarios, a multi-model collaborative mode is activated. Based on the field types and scenario relevance in the segmented files, the extraction task is assigned to the dedicated sub-model of the corresponding scenario. Each sub-model processes its domain-related fields in parallel, and then the large model aggregates and integrates the complementary information across sub-models. For fuzzy scenarios (Score < 0.6), the system automatically falls back to the general large model base. Through a flexible model scheduling mechanism, the system meets the document extraction needs of different scenarios, improving domain adaptability and extraction accuracy.

[0034] In some embodiments, obtaining document extraction results based on the fragmented files and a large model with target sub-models includes: performing document type detection on each fragmented file to determine the file attributes of each fragmented file; determining a file recognition strategy for each fragmented file based on the file attributes of each fragmented file; identifying each fragmented file as multimodal structured data based on the file recognition strategy; and inputting the multimodal structured data into a large model with target sub-models to obtain document extraction results.

[0035] In some embodiments, multimodal structured data is input into a large model with target sub-models to obtain document extraction results, including: loading the scenario-based configuration and sub-model optimization resources corresponding to the current scenario to obtain a declarative configuration file; dynamically filling the prompt word template of the target sub-model according to the declarative configuration file; if the scenario information is a single industry scenario, performing segment-by-segment iterative extraction of the multimodal structured data according to the large model dominated by the target sub-model to obtain document extraction results; if the scenario information is a mixed scenario, performing segment-by-segment iterative extraction of the multimodal structured data according to the large model with multi-target sub-model collaboration to obtain document extraction results; wherein, the large model dominated by the target sub-model performs iterative optimization according to the target sub-model; the large model with multi-target sub-model collaboration allocates collaborative tasks to each target sub-model in parallel and summarizes the extraction results of each sub-model into the large model for iterative optimization.

[0036] In this embodiment of the invention, the document extraction results are obtained based on the segmented files and the large model with target sub-models. The first step is to perform refined document type detection on each segmented file to comprehensively capture its core file attributes. The system breaks down the key features of each segment one by one: first, it determines the document format type, clarifying whether it is a text-complete, directly parsable electronic document (such as an editable PDF or Word-to-PDF conversion) or a scanned document without a text layer that requires image recognition; then, it accurately calculates the text coverage rate through algorithms, quantifying the proportion of effective text content in the document, providing a quantitative basis for subsequent identification strategy selection; at the same time, it deeply detects the content structure, identifying whether it contains special forms such as complex tables, mixed text and images, and multi-paragraph arguments; finally, it comprehensively evaluates the layout complexity, distinguishing between simple documents such as cards and invoices and complex documents such as multi-page contracts and engineering reports. The comprehensive and accurate identification of these attributes lays a solid foundation for matching the optimal processing path, ensuring that each segmented file can be processed in a targeted and efficient manner.

[0037] Based on the detected file attributes, the system will activate an intelligent decision-making mechanism to dynamically match the most suitable file recognition strategy for each file segment. If the file segment is an editable document with a complete text layer and a text coverage rate of over 95%, the fast mode will be automatically activated, directly parsing the text objects, character encodings, and PDF content streams built into the document, skipping redundant OCR recognition steps, and increasing the processing speed to 5-10 times that of the OCR mode while ensuring data accuracy. If the file segment is a scanned image with a text coverage rate of less than 80% and contains complex tables or mixed text and image structures, the system will switch to high-precision mode, simultaneously calling the OCR text recognition model and the document layout detection model. While accurately recognizing the text content, the system will also parse the page's visual structure, reorganize logical paragraphs, and construct table row and column relationships to ensure the complete extraction of complex content. If the file segment is only a simple document such as a card or invoice, the lightweight OCR mode will be activated, only calling the lightweight OCR engine to output text lines and corresponding coordinate data, minimizing computational overhead while meeting extraction requirements.

[0038] After determining the appropriate recognition strategy, the system performs deep processing on each file segment according to preset logic, transforming unstructured document data into standardized structured data containing multimodal features. In fast mode, it directly extracts the text information and precise coordinates embedded in the document, forming text-location association data. In high-precision mode, it outputs a multidimensional structured result that integrates text content, text block bounding boxes, category labels, table row and column relationships, and reorganized paragraph data, fully preserving the semantic and spatial features of the document. In OCR mode, it generates clear text line data and corresponding coordinate information, ensuring that no core information is omitted. The final output multimodal structured data not only covers basic text content but also integrates precise coordinates, semantic role labels, and page structure descriptions, providing comprehensive data support for subsequent accurate extraction, coordinate-level annotation, and cross-modal association analysis of large models.

[0039] Before inputting multimodal structured data into the large model, scenario-based configuration and resource optimization must be completed to generate a declarative configuration file adapted to the current business scenario. This configuration file, built on JSONSchema, serves as the core bridge connecting business requirements and technical implementation. It contains nine core elements: unique identifiers for extracted fields, display names, field descriptions, data types (such as strings, numbers, dates, amounts, etc.), length limits, required fields, default values, validation rule types, and post-processing methods. Simultaneously, the configuration file integrates scenario-specific keyword libraries and spatial prior knowledge from the target sub-model (such as "budget number" commonly found in the upper right corner of a financial scenario), ensuring that the configuration not only aligns with specific business needs but also provides precise support for empowering the large model from the sub-models.

[0040] Once the configuration file is loaded, the system dynamically populates and optimizes the prompt word template for the target sub-model based on its core content. The prompt word template deeply integrates field descriptions, scene keywords, validation rules, and spatial prior knowledge from the configuration file, explicitly requiring the large model to output structured results including field values, corresponding page number lists, coordinate lists, and confidence scores. The confidence score is determined by the system based on text clarity, semantic clarity, and positional rationality, providing a quantitative basis for subsequent result fusion. Through precise prompt word guidance, the large model clearly understands the extraction target, format requirements, and judgment criteria, improving the accuracy and standardization of the extraction results from the source.

[0041] If the scene recognition result is a single industry scene (scene fit score > 0.8), the large model extraction mode led by the target sub-model will be immediately activated. This mode uses a scene-specific sub-model as the core optimization unit and a general large model as the basic execution carrier. The sub-model optimizes the extraction logic of the large model in a targeted manner through a built-in scene-specific keyword library, dedicated prompt word templates, and spatial prior knowledge. Based on these scene-specific resources, the large model will perform iterative extraction of multimodal structured data segment by segment. The extraction result of each segment will carry scene semantic tags and location metadata, ensuring that the extraction logic is highly consistent with the scene business characteristics and avoiding semantic understanding biases of the general large model in vertical domains.

[0042] If the scene recognition result is a mixed scene (scene fit 0.6 ≤ Score < 0.8 and multiple candidate scenes exist), then the large model extraction mode with multi-objective sub-model collaboration is activated. The large model first acts as a task allocator, distributing extraction tasks in parallel to the corresponding scene-specific sub-models according to the field types and scene relevance in the segmented files. For example, finance-related fields are handled by the finance sub-model, and engineering-related fields are handled by the engineering sub-model. Each sub-model fully leverages its domain advantages, focusing on processing its specific domain fields to ensure domain specialization. Subsequently, the large model switches to the role of result integrator, summarizing the extraction results from each sub-model. It integrates complementary information across sub-models using scene semantic association algorithms, resolving issues such as field overlap and information dispersion, eliminating logical conflicts between the extraction results of different sub-models, and forming a logically consistent and complete initial output of segmented data. Finally, after full integration, a comprehensive and accurate document extraction result is obtained.

[0043] The following are some structural examples of large models, but they are not intended to be limiting. In actual applications, more structural settings can be made according to the scenario.

[0044] I. The large model dominated by the target sub-model (single industry scenario).

[0045] Example 1: Rule injection structure.

[0046] This structure uses a general-purpose large model (such as GPT-4 or Tongyi Qianwen) as the core execution carrier, while scenario-specific sub-models are encapsulated as "rule and resource injection modules" and do not participate in the independent semantic understanding process. The core role of the sub-models is to output scenario-specific key resources to the large model, including scenario keyword libraries (such as "budget number," "treasury payment account," and "economic classification" for fiscal scenarios, and "medical insurance catalog code" and "primary diagnosis code" for medical scenarios), spatial prior knowledge (such as key information in fiscal contracts often appearing in the upper right corner, and "surgical operation codes" concentrated in the header area of ​​the medical record homepage), and format constraint rules (such as amounts must be numeric, dates must be uniformly in "YYYY-MM-DD" format, and medical insurance codes must conform to specific length specifications). After receiving multimodal structured data, the large model completes the extraction task by combining the resources injected by the sub-models. The sub-models only intervene a second time in the result verification stage to verify the compliance of domain-specific rules. Its advantages lie in its lightweight deployment. Sub-models do not need to carry complex semantic understanding, consume low resources, and can be quickly integrated into existing large model processes, adapting to small and medium-sized business scenarios. When scenario rules change, only the injected resources of the sub-model need to be updated, without adjusting the core logic of the large model. For example, when policy adjustments cause changes in the coding rules of fiscal budget items, only the keyword library and validation rules need to be updated, shortening the iteration cycle to the hour level. It also has strong compatibility and can be adapted to various general large models without the need for customized development for specific large models, reducing dependence on technology selection.

[0047] Example 2: Pre-trained augmented architecture.

[0048] The scenario-specific sub-models are based on general large-scale model foundations (such as Llama2 and ChatGLM). They undergo secondary pre-training using scenario-specific datasets (such as 100,000 financial contracts, engineering acceptance reports, and medical records) to form a "domain-specific large-scale model" as a sub-model. This sub-model then forms a "master-slave" architecture with the original general large-scale model. The sub-models are responsible for field recognition and semantic understanding, deeply mastering domain-specific terminology, expression habits, and business logic. For example, they can accurately identify the specific references of the "five responsible parties" in engineering scenarios (construction unit, surveying unit, design unit, construction unit, and supervision unit), the calculation logic of "out-of-pocket ratio" in medical insurance scenarios, and the correspondence between "economic classification" and "budget items" in financial scenarios. The general large-scale model is responsible for general tasks such as multimodal feature association and format standardization. The two interact and collaborate in real time through an interface. The advantages of this structure are reflected in its extremely high domain precision, which can accurately handle complex expressions in the scenario (such as fuzzy semantics and professional abbreviations). For example, in the medical scenario, it can quickly associate the corresponding "primary diagnostic code" with "myocardial infarction", and in the engineering scenario, it can identify the specific calculation standard of "safe and civilized construction fee", avoiding the semantic understanding bias of general large models. It has high end-to-end efficiency, with sub-models directly leading the core extraction logic, reducing the semantic guessing links of general large models, and improving the processing speed of a single segment by more than 30%. Moreover, it can achieve deep domain adaptation without additional rule injection, and can cope with the refined extraction needs in complex scenarios, such as processing labor contracts with special clauses and complex engineering completion acceptance reports.

[0049] Example 3: Feature-enhanced structure.

[0050] In this structure, scenario-specific sub-models focus on domain-specific enhancement of multimodal features, while the general-purpose main model is responsible for the final semantic extraction and structured output. The sub-models first preprocess the multimodal structured data, filtering out scenario-related core features. For example, in a financial scenario, they prioritize extracting tabular data and filtering irrelevant images and text; in a medical scenario, they enhance temporal features (the order of diagnosis and treatment processes) and relational features (the correspondence between diagnosis and symptoms); and in an engineering scenario, they highlight the binding of text and visual features (the correlation between construction permit numbers and seal positions). The sub-models then pass the enhanced feature data to the general-purpose main model, while providing scenario-specific extraction priority rules (e.g., in a financial scenario, they prioritize extracting core financial fields such as "budget number" and "price-tax separation amount"). Its advantages include accurately filtering invalid features, reducing the processing burden on the main model, and improving extraction efficiency; enhancing core scenario features and relationships, helping the main model quickly focus on key information and reducing the risk of omissions; and adapting to scenarios rich in multimodal data, such as complex contracts containing images, tables, and seals, allowing the main model to better utilize non-textual information to assist in extraction through feature enhancement.

[0051] II. Large-scale model with multi-objective sub-model collaboration (hybrid scenario).

[0052] Example 1: Field mapping collaborative structure.

[0053] The general-purpose large model acts as a "task scheduling and result integration hub," pre-stores a "field-scenario-sub-model" mapping table, clearly defining the relationship between various fields and their corresponding scenarios and sub-models. For example, "budget number" maps to the finance sub-model, "construction permit number" maps to the engineering sub-model, "unified social credit code" maps to the market supervision sub-model, and "major diagnosis code" maps to the medical insurance sub-model. When processing mixed-scenario documents, the large model first performs preliminary identification and classification of fields in the multimodal structured data, and then allocates the extraction tasks of different domain fields in parallel to the dedicated sub-models of the corresponding scenarios according to the mapping table. Each sub-model focuses on processing the fields of its own domain and outputs extraction results including confidence scores. For example, the finance sub-model accurately extracts "treasury payment account," and the engineering sub-model extracts "supervision unit" information. After receiving the results, the large model first verifies the format standardization of the results from each sub-model, and then integrates complementary information across sub-models through scenario semantic association algorithms (such as logical consistency verification between fiscal payment amount and engineering contract amount, and matching verification between medical insurance out-of-pocket ratio and total medical expense list), ultimately forming a unified result. Its advantages lie in precise task allocation, avoiding the decrease in accuracy caused by cross-domain processing of sub-models; parallel processing improves efficiency, and synchronous work of each sub-model shortens the overall extraction time; the result integration logic is clear, and cross-domain information is ensured to be consistent through mapping tables and semantic association algorithms, making it suitable for mixed scenarios with multiple document types, such as documents that simultaneously contain financial payment application forms and engineering contract attachments.

[0054] Example 2: Hierarchical division of labor collaborative structure.

[0055] The general-purpose model plays the role of "top-level coordination and final decision-making," while multiple scenario-specific sub-models are divided according to a "domain specialization + hierarchical collaboration" model. The first layer consists of basic extraction sub-models (such as finance, engineering, personnel, and medical insurance sub-models), which handle the extraction of core fields in their respective domains. For example, the finance sub-model extracts "budget items," the personnel sub-model extracts "professional title evaluation information," and the medical insurance sub-model extracts "out-of-pocket expenses." The second layer consists of cross-validation sub-models, which are specifically responsible for handling cross-domain fields (such as common related fields like "project number," "amount," and "project leader" involved in mixed documents). They cross-validate the results of the basic extraction sub-models, for example, verifying whether the fiscal payment amount corresponding to the same "project number" is consistent with the engineering contract amount, or whether the project leader information matches the personnel file record. The main model is responsible for allocating basic extraction tasks, summarizing basic results, calling cross-validation sub-models, and resolving final conflicts based on strategies such as confidence priority and majority voting. The advantages of this structure lie in its accurate handling of cross-fields, avoiding identification biases of cross-fields by different domain sub-models through a dedicated validation sub-model; its rigorous hierarchical collaboration logic forms a closed loop of "basic extraction - cross-validation - final integration," improving result consistency; and its ability to flexibly handle complex mixed scenarios, ensuring extraction accuracy and logical coherence even in documents with many cross-domain related fields through hierarchical division of labor, such as comprehensive documents containing personnel appointments, engineering task allocations, and financial allocation plans.

[0056] Example 3: Scenario-adaptive collaborative structure.

[0057] The general-purpose large model, acting as the "scenario analysis and resource scheduling hub," first clarifies the weight and correlation of each sub-scenario in the mixed scenario based on the output of the scenario recognizer, and then dynamically allocates processing resources. The system pre-defines specific sub-models for each scenario and simultaneously constructs a "scenario association rule library" to record the field association logic between different scenarios (e.g., the mapping relationship between "project number" in the engineering scenario and "budget number" in the finance scenario, and the compliance matching between "treatment items" in the medical insurance scenario and "business items" in the market supervision scenario). During processing, the large model allocates computing resources according to the weight of the sub-scenario. Sub-models corresponding to sub-scenarios with higher weights (e.g., finance-related content accounting for 60% in the mixed document) receive more computing power and are prioritized for extraction. After each sub-model completes its specific field extraction, the large model calls the "scenario association rule library" to perform cross-scenario field association verification and integration. For example, it associates "construction unit" extracted from the engineering sub-model with "payment object" extracted from the finance sub-model through "project number" to ensure information consistency. Its advantages lie in more reasonable resource allocation, dynamically adjusting computing power according to the proportion of scenarios to improve overall processing efficiency; closer cross-scenario connections, strengthening the logical connection of fields in different scenarios through rule base to avoid information fragmentation; and adaptability to mixed documents with uneven proportions of scenarios, such as comprehensive reports that are mainly about engineering content and contain a small amount of financial payments and personnel arrangements.

[0058] In some embodiments, document type detection is performed on each fragment file to determine the file attributes of each fragment file, including: performing attribute analysis on the fragment files to obtain attribute features; wherein, the attribute features include document format type, text coverage, table features, and layout complexity.

[0059] In this embodiment of the invention, document type detection is performed on each file segment. The core of this method is to accurately characterize the file features through multi-dimensional attribute analysis, providing a basis for subsequent matching of the optimal recognition strategy. During the detection process, the system will break down the key attributes of each file segment one by one: First, determine the document format type, clarifying whether it is a text-complete, directly parsable electronic document (such as an editable PDF or Word-to-PDF conversion) or a scanned image document without a text layer that requires image recognition. This is the basis for selecting the recognition path. Then, the system will quantitatively calculate the text coverage rate through algorithms, statistically analyze the proportion of effective text content in the document, and form a quantitative indicator of text coverage. At the same time, the system will deeply detect table features, identifying whether the file segment contains tables and whether the table structure is complex (such as whether there are merged cells or multiple layers of table headers). Finally, the system will comprehensively evaluate the layout complexity, distinguishing between documents with simple structures and fixed field positions, such as cards and invoices, and documents with complex structures, such as multi-page contracts and engineering reports, containing multiple paragraphs and modules, thus comprehensively capturing the core attribute features of the file segments.

[0060] Accurate extraction of attribute features provides comprehensive support for intelligent decision-making in recognition strategies, with each feature dimension performing its specific function while working collaboratively with others. Document format directly determines whether OCR recognition is needed; editable documents require no additional image recognition step, while scanned documents must rely on OCR technology. Text coverage, as a key quantitative indicator, along with table features and layout complexity, determines the choice of recognition mode. Documents with high text coverage and no complex tables can use a fast mode to improve efficiency, while documents with low text coverage and complex tables require a high-precision mode to ensure complete extraction. Simple layout documents can balance efficiency and accuracy using a lightweight OCR mode. This multi-dimensional attribute analysis ensures the adaptability of subsequent recognition strategies to segmented files, avoiding strategy selection bias caused by single feature judgments, and laying a solid foundation for the efficiency and accuracy of the overall extraction process.

[0061] In some embodiments, the method further includes: performing multi-level validation on the extracted results based on the validation rules in the declarative configuration file, the multi-level validation including mandatory field checks, format validation and numerical reasonableness validation; performing differential post-processing on the fields according to the validation results; outputting the structured results after validation and standardization, and generating a validation report containing the validation status and failure reasons.

[0062] In this embodiment of the invention, performing multi-level verification on the extracted results based on the verification rules in the declarative configuration file is a key step in ensuring data accuracy and compliance. The system strictly follows the configuration definition and performs verification work one by one: First, it performs a mandatory field check to verify whether the fields marked as "mandatory" have been successfully extracted, preventing the loss of core information; then, it performs format verification, using regular expressions and other rules to verify whether fields such as amount, date, number, telephone, and email conform to preset format specifications, such as ensuring that the amount is numeric, the date is uniformly in the "YYYY-MM-DD" format, and the encoding meets specific length and character combination requirements; finally, it performs numerical reasonableness verification, judging whether numeric fields such as ratio and amount are within a reasonable range according to the configured numerical range rules, avoiding outliers that exceed business common sense. The three-layer verification is progressive, comprehensively filtering out all kinds of invalid information in the extracted results.

[0063] After verification, the system will perform differentiated post-processing on the fields based on different verification results, forming a closed loop of "verification-correction". For fields that pass verification, they will be standardized and transformed according to the post-processing methods set in the configuration. For example, date fields will be uniformly formatted, amount fields will have their precision controlled, and string fields will be cleaned and deduplicated to ensure the consistency of the output format. For fields that fail verification, they will be handled flexibly according to preset strategies: if required fields are missing, errors will be recorded and marked; fields with incorrect formats or values ​​outside the range can be left blank, use default values ​​as a fallback, or attempt to correct the format. For fields that are suspected of having a hallucination risk but cannot be completely determined at this time, they will be marked as "pending review" and the original value will be retained, prompting for further manual confirmation.

[0064] Ultimately, the system outputs structured results after verification and standardization, ensuring that the data received by downstream business systems is legal, accurate, and standardized. Simultaneously, a detailed verification report is generated, clearly recording the verification status of each field (pass / fail / pending review), the specific reason for failure (such as "format mismatch," "out of range," "missing required field"), and the corresponding handling method, providing clear evidence for the reliability of the extracted results. This report not only facilitates quick problem identification by business personnel but also provides complete information for audit traceability, making the entire extraction process traceable, verifiable, and controllable.

[0065] In some embodiments, the method further includes: parsing anti-hallucination rules, including data basis validity, format and scope constraints, and cross-field scenario logic consistency, based on a declarative configuration file and business logic rules in the current scenario; performing multi-dimensional verification on the extraction results according to the anti-hallucination rules and in combination with confidence assessment, identifying and marking hallucination content; implementing differentiated processing strategies according to the verification results, removing confirmed hallucination content, correcting non-fictional issues, and marking and reviewing suspected fields; and generating a prevention and control report containing hallucination detection results, a list of risk fields, and processing methods.

[0066] In this embodiment of the invention, the core starting point of the anti-illusion mechanism is to construct a multi-dimensional and comprehensive anti-illusion rule system based on the declarative configuration file and the current scenario's business logic rules. The system first deeply analyzes the field constraints defined in the declarative configuration file to clarify the basic data validity rules, including the range of required fields, data type restrictions, string length thresholds, etc.; it then organizes format and range constraint rules, such as requiring amounts to be numeric, dates to be uniformly in the "YYYY-MM-DD" format, percentage fields to be in the 0-100% range, and codes to conform to specific character combination specifications, etc. Simultaneously, it integrates scenario-specific cross-field business logic rules, such as the correspondence between "budget items" and "economic classifications" in a fiscal scenario, and the matching logic between "diagnosis codes" and "surgical codes" in a medical scenario, forming a three-layer anti-illusion rule network of "basic constraints - format specifications - logical associations," providing a clear basis for subsequent verification.

[0067] Once the rule system is built, the system will combine the confidence assessment from the extracted results to perform multi-dimensional validation on all fields, accurately identifying and marking illusory content. During the validation process, firstly, basic validity rules are used to filter out obviously invalid results such as missing required fields, inconsistent data types, and excessive length. Then, format and range constraint rules are used to filter out content with abnormal formats or ranges, such as amounts described in text, dates with inconsistent formats, or values ​​exceeding reasonable ranges. Finally, cross-field scenario logical consistency rules are used to check for extraction results with logical contradictions, such as the "Construction Permit Number" format in an engineering scenario not conforming to the preset coding rules for that scenario, or "Payment Amount" and "Budget Amount" significantly deviating in a financial scenario. Simultaneously, based on the field's confidence score, results with scores below a preset threshold and without clear original evidence are marked as high-risk for illusion, achieving accurate localization of illusory content.

[0068] After verification, the system will execute differentiated processing strategies based on the verification results and generate a detailed prevention and control report. For hallucinatory content confirmed through multiple rule verifications, such as fictitious numbers, unfounded values ​​for required fields, and speculative results that do not conform to business logic, they will be directly removed. For non-fictitious issues such as format errors and values ​​exceeding the range, corrections will be performed according to the strategies preset in the declarative configuration file, such as standardizing date formats and using default values ​​for values ​​exceeding the range. For fields suspected of being hallucinatory but not yet fully definitively determined, they will be marked as "pending review" and their original values ​​will be retained, prompting further manual verification. The final prevention and control report will clearly record the hallucination detection results, risk level, and specific handling methods for each field, providing clear support for the credibility of the extracted results and a complete basis for subsequent audit traceability and rule optimization.

[0069] In some embodiments, the method further includes: collecting candidate values, corresponding page number lists, coordinate lists, and confidence scores for each field in the document extraction results of all fragmented files after extraction by a large model dominated by the target sub-model or through collaboration of multiple target sub-models, wherein the confidence scores are determined based on a comprehensive assessment of text clarity, semantic clarity, and positional rationality; comparing multiple candidate values ​​of the same field in different fragments to identify conflict types; and executing a hierarchical conflict resolution strategy according to the conflict type to filter or splice the conflict results.

[0070] In this embodiment of the invention, collecting fragmented extraction results is the foundation for conflict resolution and fusion. The system summarizes the complete data extracted from all fragmented files by a large model dominated by the target sub-model or through collaboration of multiple target sub-models, ensuring that no candidate information is missed. The collected content not only includes the specific candidate values ​​of each field, but also synchronously associates key metadata: the page number list clarifies the distribution position of information in the original document, the coordinate list accurately records the spatial coordinates of the text on the page, providing support for subsequent mapping and annotation; while the confidence score is comprehensively judged based on text clarity (such as the degree of blurriness of text in scanned documents, the completeness of text in electronic documents), semantic clarity (such as whether the field description is unambiguous and whether it fits the business logic of the scenario), and location rationality (such as whether it conforms to the common location patterns of key information in the scenario), forming a quantitative index of 0-1, providing an objective basis for result selection.

[0071] After collecting the results, the system compares multiple candidate values ​​for the same field across different segments to accurately identify various conflict types. Common conflicts include three categories: First, inconsistent field values, meaning the content of the same field extracted from different segments differs, such as "project amount" being "500,000 yuan" in one segment and "600,000 yuan" in another; second, complementary fragments scattered across pages, meaning the field content is split across different segments due to document page breaks, such as the "contract terms" section being distributed across two segments due to table page breaks, resulting in incomplete candidate values ​​in either segment; and third, multiple identical or similar values ​​appearing repeatedly, meaning the same field is extracted repeatedly in multiple segments with identical or minimally different content, such as "unified social credit code" being extracted as the same string in all three segments.

[0072] For different conflict types identified, the system executes a hierarchical conflict resolution strategy to intelligently filter or stitch the results. For conflicts with inconsistent field values, a confidence-first strategy is adopted, directly selecting the candidate value with the highest confidence score as the optimal solution. If multiple candidate values ​​exist for the highest score, further filtering is performed based on the rationality of the position. For complementary segments scattered across pages, a cross-page merging strategy is adopted, seamlessly stitching the split segments into complete field values ​​based on page number order, coordinate relevance, and semantic coherence, ensuring that cross-page information is not broken. For recurring field values, a majority voting strategy is adopted, counting the frequency of each candidate value and selecting the candidate value with a frequency exceeding half as the final result. If no candidate value exceeds half, the optimal solution is determined by combining the confidence score and the scenario business rules of the target sub-model, ultimately forming a complete, consistent, and accurate field extraction result.

[0073] Figure 2 This is a schematic diagram of the structure of a general-purpose intelligent and reliable document extraction system based on multimodal fusion, provided in an embodiment of the present invention. Figure 2 As shown, the multimodal fusion general document intelligent and reliable extraction system includes: Module 21 is used to retrieve the original document; Parsing module 22 is used to parse the scene information of the original document and determine the target sub-model corresponding to the scene information; The segmentation module 23 is used to segment the original document using an overlapping segmentation and merging algorithm to obtain fragmented files; wherein, the segmentation parameters of the overlapping segmentation and merging algorithm are adjusted according to document features and scene features; Extraction module 24 is used to obtain document extraction results from the segmented files and the large model with target sub-models set.

[0074] Optionally, the segmentation module 23 is used to perform structural pre-analysis on the original document, calculate the title score and table score for each page, and label the page type according to the score values; determine the basic overlap strategy based on the page type; extract scene features from scene information, and determine scene weight coefficients according to the scene features to optimize and adjust the basic overlap strategy; extract document features from the original document according to the scene requirements under the current scene; determine the segment length of the segmented file according to the document features; and generate the segmented file according to the adjusted basic overlap strategy and segment length.

[0075] Optionally, the parsing module 22 is used to perform three-level feature extraction on the original document to obtain basic features; wherein, the three-level feature extraction includes metadata layer extraction, text layer extraction and visual layer extraction; based on the basic features, the scene adaptation score is calculated, and the scene information of the original document is determined based on the scene adaptation score; based on the scene information, the target sub-model is determined.

[0076] Optionally, the extraction module 24 is used to perform document type detection on each segment file to determine the file attributes of each segment file; determine the file recognition strategy for each segment file based on the file attributes of each segment file; identify each segment file as multimodal structured data according to the file recognition strategy; and input the multimodal structured data into a large model with target sub-models to obtain document extraction results.

[0077] Optionally, extraction module 24 is used to load the scenario-based configuration and sub-model optimization resources corresponding to the current scenario to obtain a declarative configuration file; dynamically fill the prompt word template of the target sub-model according to the declarative configuration file; if the scenario information type is a single industry scenario, iteratively extract the multimodal structured data piece by piece according to the large model dominated by the target sub-model to obtain the document extraction result; if the scenario information is a mixed scenario, iteratively extract the multimodal structured data piece by piece according to the large model of multi-target sub-model collaboration to obtain the document extraction result; wherein, the large model dominated by the target sub-model performs iterative optimization according to the target sub-model; the large model of multi-target sub-model collaboration allocates collaborative tasks to each target sub-model in parallel and summarizes the extraction results of each sub-model into the large model for iterative optimization.

[0078] Optionally, the extraction module 24 is used to perform attribute analysis on the segmented files to obtain attribute features; among which, the attribute features include document format type, text coverage, table features and layout complexity.

[0079] Optionally, the system also includes a verification module, which is used to: perform multi-level verification on the extracted results based on the verification rules in the declarative configuration file, including mandatory field checks, format verification, and numerical reasonableness verification; perform differential post-processing on the fields according to the verification results; output the structured results after verification and standardization, and generate a verification report containing the verification status and failure reasons.

[0080] Optionally, the system also includes an anti-hallucination module, used for: parsing anti-hallucination rules, including data validity, format and scope constraints, and cross-field scenario logic consistency, based on declarative configuration files and business logic rules in the current scenario; performing multi-dimensional verification on the extracted results according to the anti-hallucination rules and confidence assessment, identifying and marking hallucination content; implementing differentiated processing strategies based on the verification results, removing confirmed hallucination content, correcting non-fictional issues, and marking and reviewing suspected fields; and generating a prevention and control report containing hallucination detection results, a list of risk fields, and processing methods.

[0081] Optionally, the system also includes a conflict resolution module, used to: collect candidate values, corresponding page number lists, coordinate lists, and confidence scores for each field in the document extraction results of all fragmented files after extraction by a large model dominated by the target sub-model or through collaboration of multiple target sub-models. The confidence scores are determined based on a comprehensive assessment of text clarity, semantic clarity, and positional rationality. The module compares multiple candidate values ​​for the same field in different fragments to identify conflict types. Based on the conflict type, the module executes a hierarchical conflict resolution strategy to filter or combine the conflict results.

[0082] The system employs a multi-dimensional annotation framework to address the diverse information in multimodal documents, including text, images, and layout. This framework not only includes traditional field location annotations but also integrates semantic association annotations, quality level annotations, and audit trajectory annotations, forming a comprehensive four-in-one annotation system. Location annotations, based on pixel-level coordinates from OCR recognition, enable precise location of extracted fields in various document formats such as PDFs, scanned documents, and images. Semantic association annotations utilize knowledge graph technology to annotate the logical relationships between fields (e.g., the association between "contract amount" and "payment method"). Quality level annotations automatically assign A, B, and C levels based on extraction confidence: A indicates direct model extraction and successful verification; B indicates manual review is required; and C indicates questionable extraction requiring close investigation. Audit trajectory annotations comprehensively record the entire process of annotation creation, modification, and confirmation, including key information such as operator, timestamp, and modified content, providing complete evidence for compliance audits.

[0083] The meticulously designed annotation rule system ensures consistency and accuracy across different scenarios. The system includes a built-in general annotation rule library and scenario-specific rule libraries, supporting flexible configuration and dynamic updates. The general rule library covers basic specifications such as field format annotation rules (e.g., date format, amount format, ID number format), location annotation rules (e.g., annotation box size, color, and transparency standards), and conflict handling rules (e.g., annotation priority when multiple fields overlap), ensuring uniformity across scenarios. The scenario-specific rule libraries are customized for different industries such as finance, law, healthcare, and government. For example, the financial scenario strengthens the precision annotation rules for financial data, supporting accurate positioning and verification annotation of multiple decimal places; the legal scenario adds clause validity annotation rules, annotating the validity level and scope of application of core content such as "force majeure clauses" and "dispute resolution clauses"; and the healthcare scenario focuses on optimizing the privacy protection annotation rules for medical record data, encrypting and controlling access permissions for sensitive information such as patient ID numbers and contact information. All rules are presented in a visual flowchart format, allowing users to adjust rule logic through drag-and-drop operations, enabling the configuration of complex annotation rules without code development.

[0084] The collaborative mechanism of intelligent annotation and manual annotation significantly improves annotation efficiency and quality. The system integrates machine learning and active learning technologies to automate and intelligently upgrade the annotation process. The intelligent annotation stage learns from massive amounts of already annotated samples to establish a mapping model between multimodal features and annotation results. This allows it to automatically identify key fields in new documents and complete initial annotations. For highly structured documents (such as standard invoices and forms), the automatic annotation accuracy can reach over 95%, while for unstructured documents (such as contracts and reports), the accuracy exceeds 85%. For the automatic annotation results, the system initiates a multi-level verification mechanism, filtering out questionable annotations through field format verification, logical relationship verification, and cross-modal information comparison verification, triggering the manual annotation process. The manual annotation interface uses a left-right split design. The left side displays the original document and intelligent annotation results, while the right side provides annotation editing tools and rule references. Annotators can directly modify annotation positions, add annotation information, or mark points of divergence. Meanwhile, the system supports multi-person cross-annotation, allowing 2-3 annotators to independently annotate the same document. The consistency of annotations is evaluated by calculating indicators such as Kappa coefficient and IoU. Annotations with significant discrepancies will be automatically submitted to an expert annotation team for arbitration to ensure the accuracy and consistency of the final annotation results.

[0085] The full lifecycle management of labeled data maximizes its value, and the system constructs a complete chain from labeled data collection, storage, processing to application. During the data collection phase, the system supports batch import of labeled data in various formats, automatically performs data cleaning and deduplication, removing duplicate, invalid, and low-quality labels to ensure the purity of the labeled dataset. Data storage adopts a distributed database architecture, associating labeled information with original documents, extraction results, and verification reports, and supporting multi-dimensional retrieval and filtering by document type, labeling time, labeler, and scenario type. During the data processing phase, the system performs feature extraction and structure transformation on the labeled data, generating standardized labeled datasets that support export to multiple formats to adapt to the training needs of different machine learning models. The application of labeled data is mainly reflected in two aspects: First, as training samples for model optimization, the system automatically counts high-frequency erroneous labels and manually corrected examples, incorporates them into the model iteration training set, and continuously improves the model extraction accuracy through a closed-loop mechanism of "labeling-training-feedback-relabeling"; Second, as the core basis for audit traceability, auditors can use labeled data to reconstruct the entire extraction process, verify the authenticity and compliance of the extraction results, and meet the strict requirements of scenarios such as financial auditing and government supervision.

[0086] In the above embodiments, the descriptions of each embodiment have their own emphasis. Parts not detailed or described in a particular embodiment can be referred to in the relevant descriptions of other embodiments. Unless otherwise specified or in conflict with logic, the terminology and / or descriptions between different embodiments are consistent and can be referenced interchangeably. Technical features in different embodiments can be combined to form new embodiments based on their inherent logical relationships.

[0087] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A general intelligent and reliable document extraction method based on multimodal fusion, characterized in that, include: Get the original document; The scene information of the original document is parsed, and the target sub-model corresponding to the scene information is determined; The original document is segmented using an overlapping segmentation and merging algorithm to obtain fragmented files; wherein, the segmentation parameters of the overlapping segmentation and merging algorithm are adjusted according to document features and scene features; Based on the segmented files and the large model containing the target sub-model, the document extraction results are obtained.

2. The general intelligent and reliable document extraction method based on multimodal fusion according to claim 1, characterized in that, The original document is segmented using an overlapping segmentation and merging algorithm to obtain fragmented files, including: Perform a structural pre-analysis on the original document, calculate the title score and table score for each page, and label the page type according to the score values; Based on the page type, determine the basic overlap strategy; Scene features are extracted from the scene information, and scene weight coefficients are determined based on the scene features to optimize and adjust the basic overlap strategy. Based on the requirements of the current scenario, extract document features from the original document; Based on the document characteristics, determine the fragment length of the fragmented file; Based on the adjusted base overlap strategy and fragment length, generate fragment files.

3. The general intelligent and reliable document extraction method based on multimodal fusion according to claim 1, characterized in that, Parsing the scene information of the original document and determining the target sub-model corresponding to the scene information includes: Three-level feature extraction is performed on the original document to obtain basic features; wherein, the three-level feature extraction includes metadata layer extraction, text layer extraction and visual layer extraction; Based on the aforementioned basic features, a scene adaptation score is calculated, and the scene information of the original document is determined based on the scene adaptation score. Based on the scene information, determine the target sub-model.

4. The general intelligent and reliable document extraction method based on multimodal fusion according to claim 1, characterized in that, Based on the fragmented files and the large model containing the target sub-model, the document extraction results are obtained, including: Perform document type detection on each file segment to determine the file attributes of each file segment; Based on the file attributes of each file segment, determine the file identification strategy for each file segment; Based on the file recognition strategy, each fragment file is identified as multimodal structured data; The multimodal structured data is input into a large model that includes the target sub-model to obtain the document extraction results.

5. The general intelligent and reliable document extraction method based on multimodal fusion according to claim 4, characterized in that, The multimodal structured data is input into a large model that includes the target sub-model to obtain document extraction results, including: Load the scenario-specific configuration and sub-model optimization resources corresponding to the current scenario to obtain a declarative configuration file; Based on the declarative configuration file, the prompt word template of the target sub-model is dynamically populated; If the type of scene information is a single industry scene, the multimodal structured data is extracted piece by piece iteratively based on the large model dominated by the target sub-model to obtain the document extraction results; If the scene information is a mixed scene, the multimodal structured data is extracted piece by piece iteratively based on the large model of multi-objective sub-model collaboration to obtain the document extraction results; The large model, dominated by the target sub-model, is iteratively optimized based on the target sub-model. The large model, which involves collaboration among multiple target sub-models, distributes collaborative tasks in parallel to each target sub-model and aggregates the extraction results from each sub-model into the large model for iterative optimization.

6. The general intelligent and reliable document extraction method based on multimodal fusion according to claim 4, characterized in that, Perform document type detection on each file segment to determine the file attributes of each file segment, including: The fragmented files are subjected to attribute analysis to obtain attribute features; wherein, the attribute features include document format type, text coverage, table features and layout complexity.

7. The general intelligent and reliable document extraction method based on multimodal fusion according to claim 5, characterized in that, The method further includes: Based on the validation rules in the declarative configuration file, multi-level validation is performed on the extraction results. The multi-level validation includes mandatory field checks, format validation, and numerical reasonableness validation. Perform differential processing on the fields based on the validation results; Output the structured results after verification and standardization, and generate a verification report that includes the verification status and reasons for failure.

8. The general intelligent and reliable document extraction method based on multimodal fusion according to claim 7, characterized in that, The method further includes: Based on the declarative configuration file and the business logic rules in the current scenario, anti-illusion rules including data basic validity, format and range constraints, and cross-field scenario logic consistency are parsed. Based on the anti-hallucination rules and combined with confidence assessment, multi-dimensional verification is performed on the extraction results to identify and label hallucination content. Based on the verification results, a differentiated processing strategy is implemented: confirmed hallucination content is removed, non-fictional issues are corrected, and suspected fields are marked for review. Generate a prevention and control report that includes hallucination detection results, a list of risk fields, and handling methods.

9. The general intelligent and reliable document extraction method based on multimodal fusion according to claim 5, characterized in that, The method further includes: Collect the candidate values, corresponding page number list, coordinate list, and confidence score of each field in the document extraction results of all fragmented files after extraction by a large model dominated by the target sub-model or by the collaboration of multiple target sub-models. The confidence score is based on a comprehensive judgment of text clarity, semantic clarity, and positional rationality. Compare multiple candidate values ​​of the same field across different partitions to identify conflict types; A hierarchical conflict resolution strategy is executed based on the conflict type, and the conflict results are filtered or combined.

10. A multimodal fusion-based general-purpose intelligent and reliable document extraction system, characterized in that, include: The acquisition module is used to acquire the original document; The parsing module is used to parse the scene information of the original document and determine the target sub-model corresponding to the scene information; The segmentation module is used to segment the original document using an overlapping segmentation and merging algorithm to obtain fragmented files; wherein the segmentation parameters of the overlapping segmentation and merging algorithm are adjusted according to document features and scene features; The extraction module is used to obtain document extraction results from the segmented files and the large model with the target sub-model set.

Images