A hierarchical progressive document information extraction method for super-high resolution images

Abstract

The application provides a layered progressive document information extraction method and system for super-high resolution images, and in view of the problems of detail loss, content fragmentation and low extraction efficiency in the prior art when processing super-high resolution document images, the application proposes a technical solution comprising four core modules: a multi-level content-aware image segmentation module, which combines edge detection and text line positioning to decompose the super-high resolution image into sub-image blocks suitable for processing by a large model, while maintaining the integrity of key-value pair information units; a semantic retrieval module, which constructs a candidate image index based on text content recognition and field label matching; a precise extraction module, which extracts target fields only in the candidate regions; and a result integration and deduplication module, which uses a voting mechanism to solve multi-value conflicts. The application can improve the field extraction accuracy from less than 50% to more than 90%, significantly improving the efficiency and accuracy of super-high resolution document information extraction.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and intelligent document processing technology, specifically to document image processing, visual language model application, high-resolution image analysis and structured information extraction technology. It is a method and system for extracting key fields from ultra-high resolution document images using a multimodal large language model (MLLM). More specifically, it relates to a hierarchical progressive document information extraction method for ultra-high resolution images. Background Technology

[0002] With the rapid development of multimodal large language model technology, document understanding and information extraction using visual language models has become a research hotspot. Existing technologies mainly include the following approaches:

[0003] (1) Direct scaling scheme

[0004] The most direct approach is to scale the ultra-high-resolution image directly to the input size supported by the model (such as 512×512 pixels or 1024×1024 pixels). Taking a document image with a resolution of 8000×12000 pixels as an example, scaling it to 1024×1024 pixels results in a scaling ratio of approximately 1:8 to 1:12. In the original image, 6-point font (approximately 8 pixels high) is scaled down to less than 1 pixel, leading to complete loss of text information. While this method is simple and direct, it results in a significant loss of image detail, especially for documents containing small fonts, dense tables, or intricate charts. The scaled image often fails to retain sufficient identifiable information, leading to a significant decrease in recognition accuracy.

[0005] (2) Sliding window segmentation scheme

[0006] Solutions such as UReader, Monkey, and TextMonkey employ a sliding window technique to divide the input image into fixed-size image blocks. For example, UReader pre-sets a set of grid configurations with different aspect ratios and selects the grid scheme that best matches the input image for segmentation; TextMonkey segments the image into non-overlapping 448×448 pixel blocks. While these methods alleviate resolution limitations to some extent, the fixed segmentation strategy may result in the fragmentation of semantic content, affecting the model's perception and understanding capabilities.

[0007] (3) Dynamic resolution adaptation scheme

[0008] LLaVA-UHD proposes an image modularization strategy, dividing native resolution images into variable-sized slices; InternLM-XComposer2-4KHD introduces a dynamic layout arrangement strategy, which can adaptively adjust the image patch layout; the GOT model supports dynamic processing of 1024×1024 input resolution. These types of solutions usually require special design during the model training phase and are highly dependent on the model architecture.

[0009] (4) End-to-end document parsing scheme

[0010] End-to-end solutions like DeepSeek-OCR compress the number of visual tokens in high-resolution images by designing specialized visual encoders (such as DeepEncoder). These solutions also require specially trained model components and are difficult to apply directly to key field extraction from ultra-high-resolution images.

[0011] The existing technology has the following defects / deficiencies:

[0012] (1) Self-deficiency

[0013] Key information loss problem: Direct scaling can cause irreversible loss of detailed information. For documents containing dense text (e.g., more than 50 characters per square centimeter), complex tables (e.g., more than 10 rows × 10 columns), or small font content (e.g., less than 8 points), scaling often fails to accurately identify key information, with an accuracy rate usually below 40%.

[0014] Blindness of segmentation strategies: Fixed-window sliding segmentation schemes cannot perceive the semantic boundaries of document content, and may cut off complete key-value pairs, table rows or paragraphs, resulting in the loss of contextual information. Key information may be segmented into different image blocks, affecting the extraction accuracy.

[0015] Model dependency issue: Dynamic resolution adaptation and end-to-end solutions usually require specialized training or fine-tuning of the model, which cannot be directly applied to existing general-purpose multimodal large models, thus limiting the versatility and scalability of the solution.

[0016] (2) Indirect insufficiency

[0017] Efficiency issues: Existing solutions, when processing ultra-high-resolution documents, lead to an excessively large number of input tokens (e.g., exceeding 32,000 tokens) if all image content is input at once. This can exceed the model context window limit or cause a sharp drop in processing efficiency, and can also result in the "Lost in the Middle" phenomenon. If a block-by-block processing method is used, complete field extraction is required for each image block, which... Image patch extraction This method requires [number of fields]. The second largest model call generates a large number of redundant model calls. Taking the extraction of 15 fields from 20 image patches as an example, it requires 300 model calls.

[0018] Cost issues: Large-scale model calls result in significant consumption of computing resources and API call costs, which is not conducive to deployment and application in actual production environments.

[0019] (3) Other shortcomings

[0020] Insufficient generality: Existing solutions are mostly designed for specific types of documents or specific models, and are difficult to generalize to different types of documents and different multimodal large models.

[0021] Poor interpretability: Existing methods (mainly end-to-end solutions) lack the ability to visualize and debug intermediate results, making it difficult to locate and analyze the causes of extraction errors. Summary of the Invention

[0022] This invention provides a hierarchical progressive document information extraction method for ultra-high resolution images, in order to solve the technical problems existing in the prior art.

[0023] To achieve the above objectives, the present invention provides a hierarchical progressive document information extraction method for ultra-high resolution images, comprising:

[0024] S1: First-level segmentation: While ensuring information integrity, the ultra-high resolution document image is divided into several semantically complete content regions;

[0025] S2: Establish the association between image patches and target extraction fields to provide a range of candidate images for subsequent accurate extraction;

[0026] S3: For the candidate image region located by S2, perform precise extraction of the target field value;

[0027] S4: For the same field, multiple candidate values ​​are extracted from multiple image patches, integrated, and the final unique field value is output.

[0028] In one embodiment of the present invention, optionally, the optimal range of input pixel counts for the multimodal large model is defined as follows: to ,in =500,000 pixels, =1.5 million pixels, when the total number of pixels in the first-level molecular image is cut... > In step S1, after the first-level segmentation, the second-level segmentation is triggered. The second-level segmentation is a rule-based fine-grained decomposition: these regions that exceed the optimal input size are further decomposed into smaller image patches suitable for model processing, while maintaining the integrity of information as much as possible.

[0029] In one embodiment of the present invention, optionally, an overlapping region mechanism is introduced in the second-level segmentation.

[0030] The design parameters for the overlapping area include:

[0031] Overlap ratio The value range is 15%-30%;

[0032] Overlap height The calculation formula is: ,

[0033] in, The height of the segment.

[0034] The overlapping region mechanism ensures the following (1)-(3):

[0035] (1) Content located near the segmentation boundary must appear completely within at least one image patch;

[0036] (2) The probability that the field to be extracted exists completely in at least one image patch is greater than 99%;

[0037] (3) In the subsequent processing stage, duplicate extraction results generated in the overlapping area are deduplicated.

[0038] In one embodiment of the present invention, optionally, =800,000 pixels.

[0039] In one embodiment of the present invention, optionally, in step S1, the second-level segmentation adopts the following strategy: vertical segmentation based on the maximum pixel value:

[0040] This strategy preserves the original width of the image. The height of the segment remains unchanged, except for the vertical division. The calculation formula is:

[0041] ,

[0042] in, For the target number of pixels, This represents the image width.

[0043] In one embodiment of the present invention, optionally, in step S1, the second-level segmentation adopts the following strategy:

[0044] Uniform partitioning based on a grid:

[0045] The image is divided evenly according to a preset grid size, the grid parameters including the grid width. and grid height ,

[0046] Grid width The value range is 512-1024 pixels;

[0047] Grid height The value range is 512-1024 pixels;

[0048] Number of image blocks after segmentation The calculation formula is:

[0049] ,

[0050] in, Indicates rounding up. and These represent the width and height of the original image, respectively.

[0051] In one embodiment of the present invention, optionally, in step S1, the second-level segmentation adopts the following strategy: proportional segmentation:

[0052] The image is divided into segments vertically or horizontally according to a preset ratio, and the ratio configuration of each segment can be customized.

[0053] When the document has a header-body-footer structure, configure a segmentation strategy with a vertical ratio of [0.1, 0.8, 0.1], corresponding to the header area occupying 10%, the body area occupying 80%, and the footer area occupying 10%, respectively.

[0054] When the document is a two-column document, configure a splitting strategy with a horizontal ratio of [0.5, 0.5], corresponding to the left and right columns of the document respectively.

[0055] In one embodiment of the present invention, step S1 may optionally include:

[0056] S11: Edge feature extraction: Perform edge detection processing on the input document image to identify structural boundary features in the image, including table lines, separator lines, and content area boundaries;

[0057] S12: Text line localization: Using text detection technology to obtain the position coordinate information of each text line in the document image;

[0058] S13: Content Region Fusion: Based on edge features and text line position information, identify logical content regions in the document. For adjacent and semantically related regions, perform region fusion processing to avoid producing overly fragmented segmentation results.

[0059] S14: Adaptive Region Segmentation: Based on the fused content region information, determine the position of the first-level segmentation line. The segmentation line is preferentially selected at the blank gap between content regions to ensure that key-value pairs and complete table rows are not segmented into different sub-image blocks.

[0060] In one embodiment of the present invention, optionally, when locating the text line in step S12, the bounding box coordinates of the text line are represented in quadruplicate format as (x, y, w, h), where x and y represent the coordinates of the upper left corner of the bounding box, and w and h represent the width and height of the bounding box, respectively. The bounding box coordinates of the text line are used to help determine the boundary of the content area and ensure that the complete text line is not cut off during segmentation.

[0061] In one embodiment of the present invention, optionally, step S11 employs morphological operations to extract edge features, including:

[0062] S111: Create a horizontal structural element whose width is the image width divided by the horizontal kernel scaling factor. The height is 1 pixel;

[0063] S112: Create a vertical structural element whose height is the image height divided by the vertical kernel scaling factor. The width is 1 pixel;

[0064] S113: Perform erosion and dilation operations on the binarized image, respectively, with the number of iterations... The value range is 1-5.

[0065] S114: Merge horizontal and vertical line detection results by weighted average, with weighting coefficients... The value range is 0.3-0.7.

[0066] In one embodiment of the present invention, optionally, step S11 uses the Sobel operator to perform edge detection and calculates the image at... and The gradient of the direction and the formula for calculating the edge magnitude are as follows:

[0067] ,

[0068] in, and They are respectively direction and The gradient direction, the kernel size of the Sobel operator ranges from 3×3 to 7×7.

[0069] In one embodiment of the present invention, optionally, in step S13, the fusion strategy considers the following factors and their corresponding threshold parameters:

[0070] Spatial distance threshold When the vertical distance between adjacent areas is less than Integration is carried out in a timely manner, among which The value range is 20-50 pixels;

[0071] Minimum region size threshold When the area of ​​the region is smaller than At that time, it is merged with the adjacent area, where The value range is from 80×80 to 150×150 pixels;

[0072] Text line continuity determination: When the horizontal alignment deviation of text lines in adjacent areas is less than 50% of the text line height, they are determined to be continuous text areas;

[0073] The criteria for determining regional integration are expressed as follows:

[0074] ,

[0075] in, Indicates the vertical distance between adjacent regions. This indicates the area of ​​the current region.

[0076] In one embodiment of the present invention, optionally, in step S14, the selection of the cleaving line position follows the following priority rule:

[0077] First priority: Select blank gap width greater than The location, among which ≥20 pixels;

[0078] Second priority: Select locations that do not contain any text line bounding boxes;

[0079] Third priority: Select edge locations from the edge detection results;

[0080] The first level of segmentation adopts a conservative strategy, preferring to segment larger areas in order to ensure the integrity of the content. Each sub-region after segmentation contains one or more complete content units.

[0081] In one embodiment of the present invention, step S2 may optionally include:

[0082] S21: Image Text Content Recognition

[0083] For each sub-image after the first-level segmentation, extract the text content it contains for subsequent field label matching;

[0084] S22: Field Label Matching and Candidate Image Index Construction

[0085] Based on a predefined list of target fields, the identified text content is matched with the field labels, and an index of fields to candidate image patches is constructed.

[0086] In one embodiment of the present invention, step S21 may optionally include:

[0087] S211: Input size judgment:

[0088] The check determines whether the sub-image sizes after the first-level segmentation are within the optimal input range of the multimodal large model. The criteria are as follows:

[0089] like If the result is not found, proceed directly to the text content recognition step.

[0090] like Then, auxiliary secondary segmentation is needed for this sub-image;

[0091] S212: Auxiliary secondary segmentation:

[0092] For first-level segmented molecular images that are too large, a grid-based uniform segmentation strategy is used for further decomposition. The parameters for the auxiliary second-level segmentation are configured as follows:

[0093] Grid size: 768×768 pixels to 1024×1024 pixels

[0094] Overlap ratio: 20%-30%;

[0095] S213: Text Content Recognition

[0096] For sub-images of suitable size, a multimodal large model is used for text content recognition, and specially designed prompt words are used to guide the model to output all text information contained in the image;

[0097] S214: Summary of Text Content:

[0098] For images that have undergone auxiliary secondary segmentation, the text content identified by each secondary sub-block is summarized and deduplicated, and duplicate text generated by overlapping areas is removed. Finally, the summarized text content is assigned to the corresponding first-level segment sub-image, establishing a one-to-one mapping relationship between the first-level segment sub-image and its complete text content.

[0099] In one embodiment of the present invention, step S22 may optionally include:

[0100] S221: Field vocabulary construction:

[0101] Each target field is constructed with its own main label and synonyms. These labels cover various ways the field may be expressed in different documents. The field vocabulary is generated manually or through a large language model to produce synonyms for the fields to be extracted.

[0102] S222: Text normalization processing:

[0103] The text content and tag words of the first-level segmented molecular images are standardized, including removing redundant spaces and special characters, unifying capitalization, and removing special symbols to eliminate the impact of format differences on the matching results.

[0104] S223: Field tag matching:

[0105] The standardized text content is matched with the tag words of each target field.

[0106] Supports both exact match and fuzzy match modes:

[0107] Exact match: Checks if the text content completely contains the tag words, with the following matching conditions:

[0108] ,

[0109] Fuzzy matching: This method uses string similarity for more flexible matching. The similarity calculation formula is as follows:

[0110] ,

[0111] in, Represents the longest common subsequence. Indicates the string length, when A successful match is determined when the similarity threshold is reached. The value range is 0.7-0.9;

[0112] S224: Candidate Image Index Construction:

[0113] Based on the matching results, a mapping index from the target field to the first-level segmented molecular images is constructed. For each target extraction field, all first-level segmented molecular images that match the label of that field are collected, and an index structure from the field name to the candidate sub-image list is established. This index will be used for targeted processing in the extraction stage.

[0114] In one embodiment of the present invention, step S3 may optionally include:

[0115] S31: Candidate Image Screening

[0116] For each field to be extracted, the first-level segmented sub-image list corresponding to the field is obtained from the candidate image index. If no sub-image is matched for a certain field, it is marked as "not detected". Only fields with candidate images are processed for subsequent extraction.

[0117] S32: Field Extraction

[0118] For each field with candidate images, select an appropriate extraction strategy based on the size of the candidate images.

[0119] S321: Candidate image size determination:

[0120] Check the number of pixels in the first-order molecular image. Does it meet the maximum size requirement for direct input of a large multimodal model? ,

[0121] If the size meets the requirements, field extraction is performed directly on the sub-image. If the size does not meet the requirements, field extraction needs to be performed using the results of the second-level segmentation.

[0122] S322: Extraction Strategy Selection:

[0123] Direct extraction: Construct an extraction request containing the image and field names, and directly call the multimodal large model to extract field values.

[0124] Extraction based on two-level segmentation: Using the second-level segmentation results generated in the multi-level segmentation module, fields are extracted from each second-level sub-block separately;

[0125] S323: Extract Request Construction:

[0126] Candidate image patches are combined with target field names to construct specific extraction prompts, which guide the model to extract values ​​of specified fields from the image. The model then returns the extracted values ​​of the specified fields.

[0127] In one embodiment of the present invention, optionally, step S32 involves batch parallel processing of multiple extraction tasks, including the following steps:

[0128] S321': Combine multiple field-image patches into a batch request;

[0129] S322′: Utilize multithreading or asynchronous mechanisms to invoke the model in parallel;

[0130] S323′: Collect and summarize the results of each extraction task.

[0131] In one embodiment of the present invention, step S4 may optionally include:

[0132] S41: Summary of Results

[0133] Collect all results returned by extraction tasks, group and summarize them by field name, and let the field set be... For fields Its candidate value set is ;

[0134] S42: Voting Deduplication Mechanism

[0135] When there are multiple extracted values ​​for the same field, a voting mechanism is used to select the final result:

[0136] (1) Count the frequency of each value: For the field candidate values Count the number of times it appears in all the sampling results:

[0137] ,

[0138] in, This represents the total number of tasks extracted. For the first The result set of the extraction task This is an indicator function.

[0139] (2) Select the most frequent value: The value that appears most frequently is taken as the final extraction result for this field.

[0140] ,

[0141] (3) Confidence Calculation: The confidence level of the result is calculated based on the ratio of the highest frequency to the total number of extractions.

[0142] ,

[0143] in, Representation field The total number of candidate values.

[0144] S43: Output Results

[0145] The deduplicated extraction results are organized into a final structured output format, including field names, extracted values, confidence scores, and source image patch information.

[0146] The present invention also provides a hierarchical progressive document information extraction system for ultra-high resolution images, for performing the above method, comprising:

[0147] The multi-level content-aware image segmentation module is used to perform the first-level segmentation, dividing the ultra-high resolution document image into several semantically complete content regions while ensuring information integrity.

[0148] The semantic retrieval module is used to establish the association between image patches and target extraction fields, providing a range of candidate images for subsequent accurate extraction;

[0149] The precise extraction module accurately extracts the target field values ​​for the candidate image regions located by the semantic retrieval module.

[0150] The result integration and deduplication module integrates multiple candidate values ​​extracted from multiple image patches for the same field and outputs the final unique field value.

[0151] The hierarchical progressive document information extraction method and system for ultra-high resolution images provided by this invention have the following beneficial effects:

[0152] (1) Maintain the original resolution and avoid information loss.

[0153] This invention employs a multi-level segmentation strategy to decompose and process ultra-high-resolution images, maintaining the original resolution throughout the process and avoiding the loss of detail caused by scaling and downsampling. For complex documents containing small fonts, dense tables, and intricate charts, it can preserve complete and identifiable information.

[0154] (2) Content-aware segmentation to ensure information integrity

[0155] By employing a content-aware segmentation method that combines edge detection and text line localization, we ensure that key information units in a document (such as key-value pairs and table rows) are not severed at segmentation boundaries. The key-value pair integrity retention rate is improved from 50%-70% in traditional fixed-window segmentation schemes to over 95%.

[0156] (3) Separation of retrieval and extraction greatly improves efficiency.

[0157] By adopting a retrieval-extraction separation processing architecture, the permutation and combination calling pattern of "all image patches × all fields" is avoided. Taking the extraction of M fields from N image patches as an example:

[0158] Traditional solutions require N×M model calls (e.g., 20×15=300 times).

[0159] This invention only requires The model call, in which The number of candidate images for the i-th field (usually) Assuming there are an average of 2 candidate images per field, only 15 × 2 = 30 calls are needed;

[0160] The number of model calls is reduced by 80%-95%, significantly reducing computing costs and API call fees.

[0161] (4) Differentiated segmentation in stages, taking into account both identification and extraction

[0162] This invention addresses the needs of two different stages: semantic retrieval and precise extraction, by employing a differentiated two-level segmentation strategy:

[0163] The auxiliary secondary segmentation in the semantic retrieval stage uses grid segmentation, focusing only on the accuracy of text content recognition.

[0164] The second-level segmentation in the precise extraction stage adopts vertical segmentation to maintain the integrity of key-value pairs of information units to the greatest extent.

[0165] This differentiated design ensures both the integrity of text recognition and the accuracy of field extraction.

[0166] (5) No model training required, strong versatility and scalability

[0167] This invention does not rely on a specific model architecture and requires no training or fine-tuning of large multimodal models. It can be directly applied to various general-purpose large multimodal models (such as GPT-4V, Qwen-VL, InternVL, etc.). This makes the solution highly versatile and scalable, facilitating migration and application across different scenarios and models.

[0168] (6) Achieving the effect of a large model with a small model

[0169] Through the hierarchical progressive processing framework of this invention, even multimodal large models with tens of billions of parameters can achieve or even surpass the performance of models with trillions of parameters in ultra-high resolution document processing. In practical applications, extraction accuracy can be significantly improved, approaching or reaching the performance level of larger-scale models, thereby reducing model deployment costs.

[0170] (7) Flexible and configurable

[0171] Each module of this invention supports flexible configuration, including the selection of splitting strategies, the setting of overlap ratios, and the adjustment of batch processing size, which can be optimized and adjusted according to different document types and application scenarios. Attached Figure Description

[0172] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0173] Figure 1 Flowchart for semantic retrieval and candidate index construction;

[0174] Figure 2 A comparison chart of the efficiency of the retrieval-extraction separation architecture;

[0175] Figure 3 A diagram illustrating the deduplication mechanism for voting;

[0176] Figure 4 A diagram comparing content-aware segmentation and fixed-window segmentation. Detailed Implementation

[0177] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0178] Existing technologies have several drawbacks when processing ultra-high resolution document images. Direct scaling leads to loss of detail, single segmentation strategies destroy content integrity, and end-to-end extraction of large models in ultra-high resolution scenarios suffers from insufficient receptive field, loss of detail information, or context fragmentation, resulting in a significant decrease in the recognition effect of key fields. These issues make it difficult to achieve efficient and accurate extraction of key fields from complex high-resolution documents.

[0179] This invention aims to solve the following technical problems:

[0180] 1. Solve the problem of information preservation in ultra-high resolution document images: without downsampling and scaling, decompose ultra-high resolution images into sub-image blocks suitable for large model processing, while preserving the resolution and detail information of the original image.

[0181] 2. Solve the problem of content integrity during image segmentation: Through content-aware intelligent segmentation strategies, ensure that key information units in the document (such as key-value pairs, table rows, etc.) are not cut off at the segmentation boundary, and improve the key-value pair integrity retention rate from 50%-70% in traditional solutions to over 95%.

[0182] 3. Solving the efficiency problem of large model calls: By adopting a retrieval-extraction separation processing architecture, only candidate image regions related to the target field are processed in the extraction stage, reducing the number of model calls from... Reduce to The decline was 80%-95%.

[0183] 4. Provides a general solution: This solution does not require training or fine-tuning of the multimodal large model and can be directly applied to various general multimodal large models, with good versatility and scalability.

[0184] This invention proposes a hierarchical, progressive document information extraction method based on a multimodal large language model. It decomposes ultra-high-resolution document images into sub-image blocks (with the number of pixels in a single block controlled within 1.5 million) suitable for large model processing through a multi-level content-aware image segmentation strategy. A retrieval-extraction separation architecture is adopted, firstly locating candidate image regions containing target fields through semantic retrieval, and then performing precise field extraction on these candidate regions. A voting mechanism resolves multi-value conflicts, thereby achieving efficient and accurate structured information extraction while maintaining the original resolution and information integrity. The number of model calls is reduced from [previous level]. Reduce to (in For the first The number of candidate images for each field, typically ).

[0185] This invention eliminates the need for training or fine-tuning large models. Without reducing image resolution, it can improve the field extraction accuracy of models with hundreds of billions of parameters on ultra-high resolution document images from less than 50% to over 90%. At the same time, it reduces the number of large model calls by 80%-95%, significantly reducing computational costs and expanding the capabilities of small and medium-sized visual language models to process complex documents.

[0186] Figure 1 A flowchart for semantic retrieval and candidate index construction. Figure 2 A comparison chart of the efficiency of the retrieval-extraction separation architecture. Figure 3 This is a diagram illustrating the vote deduplication mechanism. Figure 4 This diagram illustrates the comparison between content-aware segmentation and fixed-window segmentation. (Example) Figures 1-4 As shown, this invention provides a hierarchical progressive document information extraction method for ultra-high resolution images, which includes:

[0187] S1: First-level segmentation: While ensuring information integrity, the ultra-high resolution document image is divided into several semantically complete content regions;

[0188] This invention adopts a layered and progressive segmentation strategy, which includes two stages: first-level content-aware segmentation and second-level fine-grained segmentation. For example, step S1 above is the first-level content-aware segmentation.

[0189] S2: Establish the association between image patches and target extraction fields to provide a range of candidate images for subsequent accurate extraction;

[0190] The goal of step S2 is to establish the association between image patches and target extraction fields, providing a range of candidate images for subsequent accurate extraction. This is one of the key differences between this invention and traditional multimodal RAG (retrieval-enhanced generation) schemes.

[0191] Differences from traditional multimodal RAG:

[0192] Traditional multimodal RAG schemes typically employ the following process: encoding images into vector representations, retrieving related images based on vector similarity, and using the retrieval results as context input to a large model to generate the answer. This approach has the following limitations:

[0193] (1) Relying on the vectorized representation of images may result in the loss of fine-grained textual semantic information;

[0194] (2) The retrieval granularity is too coarse, making it difficult to accurately locate the position of a specific field;

[0195] (3) The retrieval and generation are tightly coupled, making it difficult to perform differentiated processing for different fields;

[0196] Step S2 (semantic retrieval module) of the present invention adopts a different technical approach:

[0197] (1) Content-based retrieval: Utilizes a multimodal large model to directly understand the text content in image patches, rather than relying on vectorized representations;

[0198] (2) Label-based localization: By matching the extracted text content with predefined field labels, the image blocks in which each target field may appear can be accurately located;

[0199] (3) Decoupling of retrieval and extraction: The retrieval stage is only responsible for narrowing down the candidate range, while the extraction stage independently extracts the precise value.

[0200] S3: For the candidate image region located by S2, perform precise extraction of the target field value;

[0201] Due to the overlapping region design and multi-candidate image processing mechanism, the same field may yield multiple candidate values ​​from multiple image blocks. Step S4 (result integration and deduplication module) is responsible for integrating these candidate values ​​and outputting the final unique field value.

[0202] S4: For the same field, multiple candidate values ​​are extracted from multiple image patches, integrated, and the final unique field value is output.

[0203] The sub-regions generated by the first-level segmentation may still exceed the optimal input size range of the multimodal large model. In one embodiment of the invention, optionally, the optimal input pixel count range for the multimodal large model is defined as follows: to ,in =500,000 pixels, =1.5 million pixels, when the total number of pixels in the first-level molecular image is cut... > In step S1, after the first-level segmentation, the second-level segmentation is triggered. The second-level segmentation is a rule-based fine-grained decomposition: these regions that exceed the optimal input size are further decomposed into smaller image patches suitable for model processing, while maintaining the integrity of information as much as possible.

[0204] To further avoid potential content loss at the segmentation boundaries, this invention introduces an overlapping region mechanism in the second-level segmentation.

[0205] The design parameters for the overlapping area include:

[0206] Overlap ratio The value range is 15%-30%;

[0207] Overlap height The calculation formula is: ,

[0208] in, The height of the segment.

[0209] For example, when =1000 pixels, When =20%, =200 pixels, and similarly, overlap can also be made in the width direction.

[0210] The overlapping region mechanism ensures the following (1)-(3):

[0211] (1) Content located near the segmentation boundary must appear completely within at least one image patch;

[0212] (2) The probability that the field to be extracted exists completely in at least one image patch is greater than 99%;

[0213] (3) In the subsequent processing stage, duplicate extraction results generated in the overlapping area are deduplicated.

[0214] In one embodiment of the present invention, optionally, =800,000 pixels.

[0215] In one embodiment of the present invention, optionally, in step S1, the second-level segmentation adopts the following strategy: vertical segmentation based on the maximum pixel value:

[0216] This strategy preserves the original width of the image. The height of the segment remains unchanged, except for the vertical division. The calculation formula is:

[0217] ,

[0218] in, The target pixel count (preferably 800,000 pixels). This represents the image width.

[0219] For example, for width =2000 pixel sub-image, =800000 / 2000=400 pixels.

[0220] This approach is particularly suitable for document layouts where key-value pairs are arranged horizontally, as it maximizes the integrity of the key-value pairs.

[0221] In one embodiment of the present invention, optionally, in step S1, the second-level segmentation adopts the following strategy:

[0222] Uniform partitioning based on a grid:

[0223] The image is divided evenly according to a preset grid size, the grid parameters including the grid width. and grid height ,

[0224] Grid width The value range is 512-1024 pixels, with 768 pixels being preferred;

[0225] Grid height The value range is 512-1024 pixels, with 768 pixels being preferred;

[0226] Number of image blocks after segmentation The calculation formula is:

[0227] ,

[0228] in, Indicates rounding up. and These represent the width and height of the original image, respectively.

[0229] In one embodiment of the present invention, optionally, in step S1, the second-level segmentation adopts the following strategy: proportional segmentation:

[0230] The image is divided into segments vertically or horizontally according to a preset ratio, and the ratio configuration of each segment can be customized.

[0231] When the document has a header-body-footer structure, configure a segmentation strategy with a vertical ratio of [0.1, 0.8, 0.1], corresponding to the header area occupying 10%, the body area occupying 80%, and the footer area occupying 10%, respectively.

[0232] When the document is a two-column document, configure a splitting strategy with a horizontal ratio of [0.5, 0.5], corresponding to the left and right columns of the document respectively.

[0233] This invention employs a content-aware segmentation method that combines edge detection and text line localization. In one embodiment of this invention, step S1 optionally includes:

[0234] S11: Edge feature extraction: Perform edge detection processing on the input document image to identify structural boundary features in the image, including table lines, separator lines, and content area boundaries;

[0235] S12: Text line localization: Using text detection technology to obtain the position coordinate information of each text line in the document image;

[0236] S13: Content Region Fusion: Based on edge features and text line position information, identify logical content regions in the document. For adjacent and semantically related regions, perform region fusion processing to avoid producing overly fragmented segmentation results.

[0237] S14: Adaptive Region Segmentation: Based on the fused content region information, determine the position of the first-level segmentation line. The segmentation line is preferentially selected at the blank gap between content regions to ensure that key-value pairs and complete table rows are not segmented into different sub-image blocks.

[0238] In one embodiment of the present invention, optionally, when locating the text line in step S12, the bounding box coordinates of the text line are represented in quadruplicate format as (x, y, w, h), where x and y represent the coordinates of the upper left corner of the bounding box, and w and h represent the width and height of the bounding box, respectively. The bounding box coordinates of the text line are used to help determine the boundary of the content area and ensure that the complete text line is not cut off during segmentation.

[0239] In one embodiment of the present invention, optionally, step S11 employs morphological operations to extract edge features, including:

[0240] S111: Create a horizontal structural element whose width is the image width divided by the horizontal kernel scaling factor. (Value range is 30-100, preferably 50), height is 1 pixel;

[0241] S112: Create a vertical structural element whose height is the image height divided by the vertical kernel scaling factor. (Value range is 50-150, preferably 100), width is 1 pixel;

[0242] S113: Perform erosion and dilation operations on the binarized image, respectively, with the number of iterations... The value range is 1-5 times, with 3 times being preferred;

[0243] S114: Merge horizontal and vertical line detection results by weighted average, with weighting coefficients... The value range is 0.3-0.7, with 0.5 being preferred.

[0244] In one embodiment of the present invention, optionally, step S11 uses the Sobel operator to perform edge detection and calculates the image at... and The gradient of the direction and the formula for calculating the edge magnitude are as follows:

[0245] ,

[0246] in, and They are respectively direction and The gradient direction, the kernel size of the Sobel operator ranges from 3×3 to 7×7, with 3×3 being preferred.

[0247] In one embodiment of the present invention, optionally, in step S13, the fusion strategy considers the following factors and their corresponding threshold parameters:

[0248] Spatial distance threshold When the vertical distance between adjacent areas is less than Integration is carried out in a timely manner, among which The value range is 20-50 pixels, with 30 pixels being preferred;

[0249] Minimum region size threshold When the area of ​​the region is smaller than At that time, it is merged with the adjacent area, where The value range is from 80×80 to 150×150 pixels, with 100×100 pixels being preferred;

[0250] Text line continuity determination: When the horizontal alignment deviation of text lines in adjacent areas is less than 50% of the text line height, they are determined to be continuous text areas;

[0251] The criteria for determining regional integration are expressed as follows:

[0252] ,

[0253] in, Indicates the vertical distance between adjacent regions. This indicates the area of ​​the current region.

[0254] In one embodiment of the present invention, optionally, in step S14, the selection of the cleaving line position follows the following priority rule:

[0255] First priority: Select blank gap width greater than The location, among which ≥20 pixels;

[0256] Second priority: Select locations that do not contain any text line bounding boxes;

[0257] Third priority: Select edge locations from the edge detection results;

[0258] The first level of segmentation adopts a conservative strategy, preferring to segment larger areas in order to ensure the integrity of the content. Each sub-region after segmentation contains one or more complete content units.

[0259] In one embodiment of the present invention, step S2 may optionally include:

[0260] S21: Image Text Content Recognition

[0261] For each sub-image after the first-level segmentation, extract the text content it contains for subsequent field label matching;

[0262] S22: Field Label Matching and Candidate Image Index Construction

[0263] Based on a predefined list of target fields, the identified text content is matched with the field labels, and an index of fields to candidate image patches is constructed.

[0264] In one embodiment of the present invention, step S21 may optionally include:

[0265] S211: Input size judgment:

[0266] The check determines whether the sub-image sizes after the first-level segmentation are within the optimal input range of the multimodal large model. The criteria are as follows:

[0267] like If the number of pixels is ≤1.5 million, then proceed directly to the text content recognition step.

[0268] like Then, auxiliary secondary segmentation is needed for this sub-image;

[0269] S212: Auxiliary secondary segmentation:

[0270] For first-level segmented molecular images that are too large, a grid-based uniform segmentation strategy is used for further decomposition. The parameters for the auxiliary second-level segmentation are configured as follows:

[0271] Grid size: 768×768 pixels to 1024×1024 pixels

[0272] Overlap ratio: 20%-30%, preferably 25%;

[0273] The second-level segmentation here does not need to consider the integrity of key-value pairs, because the goal of this stage is only to recognize the text content, not to extract field values. Even if some text is segmented, it is acceptable as long as the text can be completely recognized in a certain image patch.

[0274] S213: Text Content Recognition

[0275] For sub-images of suitable size, a multimodal large model is used for text content recognition. Specially designed prompt words guide the model to output all text information contained in the image; an example prompt word template is as follows:

[0276] Please carefully observe this image and identify and extract all visible text content within it. Output the text in top-to-bottom and left-to-right order, maintaining the original layout. If the image contains tables, headings, or labels, please identify them as well. Output in plain text format.

[0277] S214: Summary of Text Content:

[0278] For images that have undergone auxiliary secondary segmentation, the text content identified by each secondary sub-block is summarized and deduplicated, and duplicate text generated by overlapping areas is removed. Finally, the summarized text content is assigned to the corresponding first-level segment sub-image, establishing a one-to-one mapping relationship between the first-level segment sub-image and its complete text content.

[0279] In one embodiment of the present invention, step S22 may optionally include:

[0280] S221: Field vocabulary construction:

[0281] Each target field is constructed with its own main label and synonyms. These labels cover various ways the field may be expressed in different documents. The field vocabulary is generated manually or through a large language model to produce synonyms for the fields to be extracted.

[0282] S222: Text normalization processing:

[0283] The text content and tag words of the first-level segmented molecular images are standardized, including removing redundant spaces and special characters, unifying capitalization, and removing special symbols to eliminate the impact of format differences on the matching results.

[0284] S223: Field tag matching:

[0285] The standardized text content is matched with the tag words of each target field.

[0286] Supports both exact match and fuzzy match modes:

[0287] Exact match: Checks if the text content completely contains the tag words, with the following matching conditions:

[0288] ,

[0289] Fuzzy matching: This method uses string similarity for more flexible matching. The similarity calculation formula is as follows:

[0290] ,

[0291] in, Represents the longest common subsequence. Indicates the string length, when A successful match is determined when the similarity threshold is reached. The value range is 0.7-0.9, with 0.8 being preferred;

[0292] S224: Candidate Image Index Construction:

[0293] Based on the matching results, a mapping index from the target field to the first-level segmented molecular images is constructed. For each target extraction field, all first-level segmented molecular images that match the label of that field are collected, and an index structure from the field name to the candidate sub-image list is established. This index will be used for targeted processing in the extraction stage.

[0294] Through the above process, a complete semantic retrieval process from image text content recognition to candidate image index construction was completed, laying the foundation for subsequent accurate extraction.

[0295] In one embodiment of the present invention, step S3 may optionally include:

[0296] S31: Candidate Image Screening

[0297] For each field to be extracted, the first-level segmented sub-image list corresponding to the field is obtained from the candidate image index. If no sub-image is matched for a certain field, it is marked as "not detected". Only fields with candidate images are processed for subsequent extraction.

[0298] S32: Field Extraction

[0299] For each field with candidate images, select an appropriate extraction strategy based on the size of the candidate images.

[0300] S321: Candidate image size determination:

[0301] Check the number of pixels in the first-order molecular image. Does it meet the maximum size requirement for direct input of a large multimodal model? ,

[0302] If the size meets the requirements, field extraction is performed directly on the sub-image. If the size does not meet the requirements, field extraction needs to be performed using the results of the second-level segmentation.

[0303] S322: Extraction Strategy Selection:

[0304] Direct extraction: Construct an extraction request containing the image and field names, and directly call the multimodal large model to extract field values.

[0305] Extraction based on two-level segmentation: Using the second-level segmentation results generated in the multi-level segmentation module (Note: The second-level segmentation scheme here depends on the specific situation of the document. For example, if the key-value pairs are arranged horizontally, vertical segmentation based on the maximum pixel value is preferred to ensure information integrity), field extraction is performed on each second-level sub-block.

[0306] S323: Extract Request Construction:

[0307] Candidate image patches (first-level or second-level segmentation results) are combined with target field names to construct specific extraction prompts that guide the model to extract values ​​for the specified fields from the image. Examples of optional prompts are provided below.

[0308] Please extract the value of the "{field_name}" field from the image. The field description is "{description}", and possible synonyms for this field include "{synonyms}". Output the result in JSON format: {"field_name": "Field name", "field_value": "Extracted value"}.

[0309] The prompts clearly specify the names of the fields to be extracted and their specific meanings, improving the targeting of the extraction.

[0310] The model returns the extracted value for the specified field.

[0311] In one embodiment of the present invention, optionally, step S32 involves batch parallel processing of multiple extraction tasks, including the following steps:

[0312] S321': Combine multiple field-image patches into a batch request;

[0313] S322′: Utilize multithreading or asynchronous mechanisms to invoke the model in parallel;

[0314] S323′: Collect and summarize the results of each extraction task.

[0315] In one embodiment of the present invention, step S4 may optionally include:

[0316] S41: Summary of Results

[0317] Collect all results returned by extraction tasks, group and summarize them by field name, and let the field set be... For fields Its candidate value set is ;

[0318] S42: Voting Deduplication Mechanism

[0319] When there are multiple extracted values ​​for the same field, a voting mechanism is used to select the final result:

[0320] (1) Count the frequency of each value: For the field candidate values Count the number of times it appears in all the sampling results:

[0321] ,

[0322] in, This represents the total number of tasks extracted. For the first The result set of the extraction task This is an indicator function.

[0323] (2) Select the most frequent value: The value that appears most frequently is taken as the final extraction result for this field.

[0324] ,

[0325] (3) Confidence Calculation: The confidence level of the result is calculated based on the ratio of the highest frequency to the total number of extractions.

[0326] ,

[0327] in, Representation field The total number of candidate values.

[0328] S43: Output Results

[0329] The deduplicated extraction results are organized into a final structured output format, including field names, extracted values, confidence scores, and source image patch information.

[0330] The present invention also provides a hierarchical progressive document information extraction system for ultra-high resolution images, for performing the above method, comprising:

[0331] The multi-level content-aware image segmentation module is used to perform the first-level segmentation, dividing the ultra-high resolution document image into several semantically complete content regions while ensuring information integrity.

[0332] The semantic retrieval module is used to establish the association between image patches and target extraction fields, providing a range of candidate images for subsequent accurate extraction;

[0333] The precise extraction module accurately extracts the target field values ​​for the candidate image regions located by the semantic retrieval module.

[0334] The result integration and deduplication module integrates multiple candidate values ​​extracted from multiple image patches for the same field and outputs the final unique field value. Specific Implementation

[0335] Example 1: Precise Field Extraction from Ultra-High Resolution Documents

[0336] Application Scenario: This example uses the extraction of key fields from an ultra-high resolution contract document. The contract document image to be processed has a resolution of 6000×8000 pixels (48 million pixels in total) and contains 15 fields to be extracted, including the name of Party A, the name of Party B, the contract amount, the signing date, and the contract number.

[0337] Technical challenges: Ultra-high resolution results in rich document details but is difficult to process; direct scaling will make small fonts and dense content unrecognizable; fixed window segmentation may cut off key information units; and traditional solutions have low accuracy.

[0338] Processing flow:

[0339] Step 1: Content-aware segmentation while maintaining original resolution

[0340] (1) Edge feature extraction parameter configuration:

[0341] Create a horizontal structural element with a width of 6000 / 50 = 120 pixels (horizontal kernel ratio). The preferred value is 50), and the height is 1 pixel.

[0342] Create a vertical structural element with a height of 8000 / 100 = 80 pixels (vertical kernel scale factor). The preferred value is 100), and the width is 1 pixel.

[0343] Perform erosion and dilation operations on the binarized image, with a number of iterations. Second-rate.

[0344] By merging the detection results of horizontal and vertical lines using a weighted average, the weighting coefficients are... .

[0345] (2) Text line localization and boundary recognition:

[0346] Use OCR text detection technology to obtain the position coordinates of all text lines in a document.

[0347] Text line bounding boxes are in quadruplet format. express.

[0348] Approximately 180 lines of text and 12 main content areas were identified.

[0349] (3) Content area fusion processing:

[0350] Set spatial distance threshold Pixel.

[0351] Set minimum area size threshold Pixel.

[0352] Merge adjacent regions with a vertical distance of less than 30 pixels and consecutive text lines.

[0353] Merge small regions with an area of ​​less than 10,000 square pixels with adjacent regions.

[0354] After fusion, eight semantically complete content regions are obtained.

[0355] (4) Adaptive region partitioning results:

[0356] Select a position where the width of the blank gap between the content areas is greater than 20 pixels as the dividing line.

[0357] Prioritize locations that do not contain any text line bounding boxes.

[0358] The first-level segmentation produces eight sub-images, ranging in size from 1500×2000 pixels to 6000×3000 pixels.

[0359] All key-value pairs are preserved intact, with no critical information being cut off.

[0360] Step 2: Accurate Text Content Recognition

[0361] (1) Sub-image size evaluation:

[0362] Evaluate the number of pixels in the 8 first-level sub-images:

[0363] Sub-image 1: 1500×2000 = 3 million pixels (more than) =1.5 million)

[0364] Sub-image 2: 6000×3000 = 18 million pixels (exceeding) =1.5 million)

[0365] Sub-image 3: 6000×1500=9 million pixels (more than) =1.5 million)

[0366] Sub-images 4-8: The number of pixels is between 500,000 and 1,500,000, meeting the conditions for direct processing.

[0367] (2) Auxiliary secondary segmentation:

[0368] Perform grid partitioning on sub-images 1, 2, and 3.

[0369] The grid size is set to 768×768 pixels.

[0370] The overlap ratio is set to 25%, and the overlap width and height are both 768×0.25=192 pixels.

[0371] Sub-image 2 (6000×3000 pixels) is divided into:

[0372] Horizontal direction: piece.

[0373] Vertical direction: piece.

[0374] A total of 11 × 6 = 66 second-level sub-blocks were generated.

[0375] (3) Text content recognition:

[0376] Text recognition is performed directly on sub-images 4-8 (5 in total), resulting in 5 model calls.

[0377] Text recognition is performed in batches on the secondary sub-blocks of sub-images 1, 2, and 3:

[0378] The batch size is set to 10, and the model is called in parallel for each batch.

[0379] Sub-images 1, 2, and 3 together generate approximately 120 secondary sub-blocks.

[0380] Using batch processing mode, the actual model call batch is: Approval.

[0381] In total, the model was called in approximately 17 batches (5+12) during the text recognition stage.

[0382] (4) Summary of text content:

[0383] The recognition results of the second-level sub-blocks are summarized according to the first-level sub-images.

[0384] The text is removed from overlapping areas, resulting in a deduplication rate of approximately 15%.

[0385] Establish complete text content mappings for the eight first-level sub-images.

[0386] Step 3: Precise field label matching

[0387] (1) Construction of field vocabulary:

[0388] Build a tag vocabulary for 15 target fields, example:

[0389] "Party A Name": ["Party A", "Party A Company", "Client", "Contracting Party"].

[0390] "Contract Amount": ["Contract Amount", "Total Price", "Total Contract Price", "Amount"].

[0391] "Signing Date": ["Signing Date", "Signing Date", "Date", "Signing Time"].

[0392] On average, each field is configured with 3-5 synonym tags.

[0393] (2) Text standardization processing:

[0394] Remove extra spaces and special characters.

[0395] Standardize the format of Chinese and English characters.

[0396] The standardized text content is approximately 12,000 characters long.

[0397] (3) Field label matching:

[0398] A combination of exact matching and fuzzy matching is used, with a similarity threshold. .

[0399] Matching results statistics:

[0400] "Party A Name": Matched sub-image 1 and sub-image 2 (2 candidates).

[0401] "Party B's Name": Matched sub-image 1 and sub-image 2 (2 candidates).

[0402] "Contract Amount": Matched to sub-images 3 and 5 (2 candidates).

[0403] "Signing Date": Matched sub-images 6 and 7 (2 candidates).

[0404] "Contract Number": Matched to sub-image 1 (1 candidate).

[0405] The remaining 10 fields: on average, each field matched 2-3 candidate sub-images.

[0406] (4) Candidate image index construction:

[0407] Establish a mapping relationship between 15 fields and candidate sub-images.

[0408] Total number of candidate sub-images: 15 × 2 ≈ 30 candidate relationships.

[0409] Step 4: Targeted and precise field extraction

[0410] (1) Candidate image size determination and secondary segmentation:

[0411] Evaluate the size of the candidate sub-images to determine whether secondary segmentation is needed.

[0412] Sub-image 2 (6000×3000=18 million pixels) needs to be vertically segmented:

[0413] Maintain width =6000 pixels unchanged.

[0414] Calculate the maximum height =800000 / 600≈133 pixels.

[0415] The actual segmentation is into blocks with a height of approximately 1000 pixels (this may be appropriately relaxed to ensure content integrity).

[0416] Set the overlap ratio to 20%, and the overlap height... =1000×0.2=200 pixels.

[0417] generate A vertically segmented block.

[0418] (2) Field extraction execution:

[0419] Extract all candidate images from 15 fields.

[0420] Use batch processing mode and set the batch size to 8.

[0421] Actual number of tasks drawn: approximately 30 (15 x 2).

[0422] Model call batch: Approval.

[0423] Compared to the traditional approach (8×15=120 times), the number of model calls is reduced by 75%.

[0424] (3) Example of extraction results:

[0425] The "Name of Party A" is extracted from sub-image 1 as: ["XXX Technology Co., Ltd".

[0426] The "Name of Party A" is extracted from sub-image 2 as: ["XXX Technology Co., Ltd".

[0427] The "Name of Party A" is extracted from sub-image 2 as: ["Location"].

[0428] The "Contract Amount" is extracted from sub-image 3 as: ["1,280,000 yuan"].

[0429] The "Contract Amount" is extracted from sub-image 5 as: ["1.28 million yuan"].

[0430] Step 5: Multi-value voting and precise deduplication

[0431] (1) Voting statistics:

[0432] Candidate values ​​for "Party A Name":

[0433] "XXX Technology Co., Ltd." appeared twice.

[0434] Vote for: "XXX Technology Co., Ltd."

[0435] Confidence level calculation: 2 / 3 = 66.6%.

[0436] Candidate values ​​for "Contract Amount":

[0437] "1,280,000 yuan": appeared once

[0438] "1.28 million yuan": appeared once

[0439] After standardization, they are identified as the same value.

[0440] Voting option: "1,280,000 yuan" (Select a more detailed format)

[0441] Confidence level calculation: 2 / 2 = 100%.

[0442] (2) Final output result:

[0443] 15 fields were successfully extracted.

[0444] All field values ​​have passed multi-candidate validation, with no omissions or errors.

[0445] Example 2: Efficient batch processing of multi-page PDF documents.

[0446] Application scenario: Processing a 20-page tender document PDF, each page being A4 size (…). The scan resolution is 300 dpi, and the converted image size per page is 2480×3508 pixels (approximately 8.7 million pixels). It is necessary to extract 25 predefined fields from all pages, including project name, tender number, bid deadline, technical requirements, and evaluation criteria. These fields may be scattered across different pages.

[0447] Technical challenges: The total data volume of multi-page documents is large (20 × 8.7 million = 174 million pixels). Traditional solutions require complete field extraction of all image blocks on each page, resulting in extremely low efficiency. The present invention processes multi-page documents in a manner that is essentially the same as that of single-page documents, and has a significant advantage in efficiency.

[0448] Processing flow:

[0449] Step 1: Global Parallel Splitting

[0450] After converting the 20-page PDF into images, the same segmentation parameters as in Example 1 were used. Content-aware segmentation (using pixels, etc.) is performed on all pages simultaneously, generating a total of approximately 105 first-level sub-images.

[0451] Step 2: Global Text Recognition and Index Building

[0452] (1) Size evaluation of 105 sub-images:

[0453] The 82 sub-images, with pixel counts ranging from 500,000 to 1,500,000, are used for direct text recognition.

[0454] The 23 sub-images, each exceeding 1.5 million pixels, were segmented into grids (768×768 pixels, with 25% overlap) for identification.

[0455] (2) Field label matching and index construction:

[0456] Build a tag vocabulary for 25 target fields and set similarity thresholds. .

[0457] Perform global label matching on the text content of all first-level sub-images.

[0458] Example of matching results:

[0459] "Project Name": 2 candidate sub-images were matched.

[0460] "Tender Number": 2 candidate sub-images were matched.

[0461] "Technical Requirements": Match 11 candidate sub-images (across pages 8-12).

[0462] "Scoring criteria": Matched 8 candidate sub-images (across pages 15-17).

[0463] Approximately 90 candidate relationships were matched across 25 fields, with an average of 3.6 candidate sub-images per field.

[0464] Step 3: Batch Parallel Precise Extraction

[0465] (1) Candidate image processing:

[0466] For the 15 large candidate sub-images (approximately 17%) that require vertical segmentation, perform vertical segmentation (width) =2480 pixels unchanged, segment height 800 pixels, overlap 20%).

[0467] A total of 120 extraction tasks (75 without splitting + 45 after splitting).

[0468] (2) Batch extraction execution:

[0469] Batch size set to 15, actual batches called: batch

[0470] Compared to the traditional approach (105×25=2625 model calls), the proposed solution requires only 120 calls, reducing the number of calls by approximately 95.4% (2625-120) / 2625.

[0471] Step 4: Integrate and deduplicate cross-page results

[0472] (1) Example of deduplication in voting:

[0473] The "Project Name" was selected from two candidates:

[0474] "A certain smart city construction project": appeared twice.

[0475] "City Construction Project": Appeared once (identification error).

[0476] The vote selection is "a certain smart city construction project", with a confidence level of 2 / 3 = 66.7%.

[0477] The "Technical Requirements" section selected 10 technical requirements from 11 candidates, and after deduplication, retained 8 unique technical requirements.

[0478] The "scoring criteria" extracts scoring criteria items from 8 candidates, identifies 5 main scoring dimensions, and integrates them into a complete hierarchical structure of scoring criteria in page order.

[0479] (2) Final output result:

[0480] 25 fields were successfully extracted.

[0481] Cross-page fields are fully integrated with no missing information.

[0482] The hierarchical progressive document information extraction method and system for ultra-high resolution images provided by this invention have the following beneficial effects:

[0483] (1) Maintain the original resolution and avoid information loss.

[0484] This invention employs a multi-level segmentation strategy to decompose and process ultra-high-resolution images, maintaining the original resolution throughout the process and avoiding the loss of detail caused by scaling and downsampling. For complex documents containing small fonts, dense tables, and intricate charts, it can preserve complete and identifiable information.

[0485] (2) Content-aware segmentation to ensure information integrity

[0486] By employing a content-aware segmentation method that combines edge detection and text line localization, we ensure that key information units in a document (such as key-value pairs and table rows) are not severed at segmentation boundaries. The key-value pair integrity retention rate is improved from 50%-70% in traditional fixed-window segmentation schemes to over 95%.

[0487] (3) Separation of retrieval and extraction greatly improves efficiency.

[0488] By adopting a retrieval-extraction separation processing architecture, the permutation and combination calling pattern of "all image patches × all fields" is avoided. Taking the extraction of M fields from N image patches as an example:

[0489] Traditional solutions require N×M model calls (e.g., 20×15=300 times).

[0490] This invention only requires The model call, in which The number of candidate images for the i-th field (usually) Assuming there are an average of 2 candidate images per field, only 15 × 2 = 30 calls are needed;

[0491] The number of model calls is reduced by 80%-95%, significantly reducing computing costs and API call fees.

[0492] (4) Differentiated segmentation in stages, taking into account both identification and extraction

[0493] This invention addresses the needs of two different stages: semantic retrieval and precise extraction, by employing a differentiated two-level segmentation strategy:

[0494] The auxiliary secondary segmentation in the semantic retrieval stage uses grid segmentation, focusing only on the accuracy of text content recognition.

[0495] The second-level segmentation in the precise extraction stage adopts vertical segmentation to maintain the integrity of key-value pairs of information units to the greatest extent.

[0496] This differentiated design ensures both the integrity of text recognition and the accuracy of field extraction.

[0497] (5) No model training required, strong versatility and scalability

[0498] This invention does not rely on a specific model architecture and requires no training or fine-tuning of large multimodal models. It can be directly applied to various general-purpose large multimodal models (such as GPT-4V, Qwen-VL, InternVL, etc.). This makes the solution highly versatile and scalable, facilitating migration and application across different scenarios and models.

[0499] (6) Achieving the effect of a large model with a small model

[0500] Through the hierarchical progressive processing framework of this invention, even multimodal large models with tens of billions of parameters can achieve or even surpass the performance of models with trillions of parameters in ultra-high resolution document processing. In practical applications, extraction accuracy can be significantly improved, approaching or reaching the performance level of larger-scale models, thereby reducing model deployment costs.

[0501] (7) Flexible and configurable

[0502] Each module of this invention supports flexible configuration, including the selection of splitting strategies, the setting of overlap ratios, and the adjustment of batch processing size, which can be optimized and adjusted according to different document types and application scenarios.

[0503] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of one embodiment, and the modules or processes shown in the drawings are not necessarily essential for implementing the present invention.

[0504] Those skilled in the art will understand that the modules in the apparatus of the embodiments can be distributed in the apparatus of the embodiments as described in the embodiments, or they can be located in one or more devices different from this embodiment with corresponding changes. The modules of the above embodiments can be combined into one module, or they can be further divided into multiple sub-modules.

[0505] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A hierarchical progressive document information extraction method for ultra-high resolution images, characterized in that, include: S1: First-level segmentation: While ensuring information integrity, the ultra-high resolution document image is divided into several semantically complete content regions; S2: Establish the association between image patches and target extraction fields to provide a range of candidate images for subsequent accurate extraction; S3: For the candidate image region located by S2, perform precise extraction of the target field value; S4: For the same field, multiple candidate values ​​are extracted from multiple image patches, integrated, and the final unique field value is output.

2. The hierarchical progressive document information extraction method for ultra-high resolution images according to claim 1, characterized in that, Define the optimal range of input pixel counts for a multimodal large model as follows: to ,in =500,000 pixels, =1.5 million pixels, when the total number of pixels in the first-level molecular image is cut... > In step S1, after the first-level segmentation, the second-level segmentation is triggered. The second-level segmentation is a rule-based fine-grained decomposition: these regions that exceed the optimal input size are further decomposed into smaller image patches suitable for model processing, while maintaining the integrity of information as much as possible.

3. The hierarchical progressive document information extraction method for ultra-high resolution images according to claim 2, characterized in that, The second-level segmentation introduces an additional overlapping region mechanism. The design parameters for the overlapping area include: Overlap ratio The value range is 15%-30%; Overlap height The calculation formula is: , in, The height of the segment. The overlapping region mechanism ensures the following (1)-(3): (1) Content located near the segmentation boundary must appear completely within at least one image patch; (2) The probability that the field to be extracted exists completely in at least one image patch is greater than 99%; (3) In the subsequent processing stage, duplicate extraction results generated in the overlapping area are deduplicated.

4. The hierarchical progressive document information extraction method for ultra-high resolution images according to claim 2, characterized in that, =800,000 pixels.

5. The hierarchical progressive document information extraction method for ultra-high resolution images according to claim 2, characterized in that, In step S1, the second-level segmentation adopts the following strategy: vertical segmentation based on the maximum pixel value: This strategy preserves the original width of the image. The height of the segment remains unchanged, except for the vertical division. The calculation formula is: ， in, For the target number of pixels, This represents the image width.

6. The hierarchical progressive document information extraction method for ultra-high resolution images according to claim 2, characterized in that, In step S1, the second-level segmentation adopts the following strategy: Uniform partitioning based on a grid: The image is divided evenly according to a preset grid size, the grid parameters including the grid width. and grid height , Grid width The value range is 512-1024 pixels; Grid height The value range is 512-1024 pixels; Number of image blocks after segmentation The calculation formula is: ， in, Indicates rounding up. and These represent the width and height of the original image, respectively.

7. The hierarchical progressive document information extraction method for ultra-high resolution images according to claim 2, characterized in that, In step S1, the second-level segmentation adopts the following strategy: proportional segmentation: The image is divided into segments vertically or horizontally according to a preset ratio, and the ratio configuration of each segment can be customized. When the document has a header-body-footer structure, configure a segmentation strategy with a vertical ratio of [0.1, 0.8, 0.1], corresponding to the header area occupying 10%, the body area occupying 80%, and the footer area occupying 10%, respectively. When the document is a two-column document, configure a splitting strategy with a horizontal ratio of [0.5, 0.5], corresponding to the left and right columns of the document respectively.

8. The hierarchical progressive document information extraction method for ultra-high resolution images according to claim 1, characterized in that, Step S1 includes: S11: Edge feature extraction: Perform edge detection processing on the input document image to identify structural boundary features in the image, including table lines, separator lines, and content area boundaries; S12: Text line localization: Using text detection technology to obtain the position coordinate information of each text line in the document image; S13: Content Region Fusion: Based on edge features and text line position information, identify logical content regions in the document. For adjacent and semantically related regions, perform region fusion processing to avoid producing overly fragmented segmentation results. S14: Adaptive Region Segmentation: Based on the fused content region information, determine the position of the first-level segmentation line. The segmentation line is preferentially selected at the blank gap between content regions to ensure that key-value pairs and complete table rows are not segmented into different sub-image blocks.

9. The hierarchical progressive document information extraction method for ultra-high resolution images according to claim 7, characterized in that, In step S12, when locating the text line, the bounding box coordinates of the text line are represented in quadruple format as (x, y, w, h), where x and y represent the coordinates of the upper left corner of the bounding box, and w and h represent the width and height of the bounding box, respectively. The bounding box coordinates of the text line are used to help determine the boundary of the content area and ensure that the complete text line is not cut off during segmentation.

10. The hierarchical progressive document information extraction method for ultra-high resolution images according to claim 8, characterized in that, Step S11 uses morphological operations to extract edge features, including: S111: Create a horizontal structural element whose width is the image width divided by the horizontal kernel scaling factor. The height is 1 pixel; S112: Create a vertical structural element whose height is the image height divided by the vertical kernel scaling factor. The width is 1 pixel; S113: Perform erosion and dilation operations on the binarized image, respectively, with the number of iterations... The value range is 1-5. S114: Merge horizontal and vertical line detection results by weighted average, with weighting coefficients... The value range is 0.3-0.

7.

11. The hierarchical progressive document information extraction method for ultra-high resolution images according to claim 8, characterized in that, Step S11 uses the Sobel operator for edge detection, calculating the image's edge position. and The gradient of the direction and the formula for calculating the edge magnitude are as follows: , in, and They are respectively direction and The gradient direction, the kernel size of the Sobel operator ranges from 3×3 to 7×7.

12. The hierarchical progressive document information extraction method for ultra-high resolution images according to claim 8, characterized in that, In step S13, the fusion strategy considers the following factors and their corresponding threshold parameters: Spatial distance threshold When the vertical distance between adjacent areas is less than Integration is carried out in a timely manner, among which The value range is 20-50 pixels; Minimum region size threshold When the area of ​​the region is smaller than At that time, it is merged with the adjacent area, where The value range is from 80×80 to 150×150 pixels; Text line continuity determination: When the horizontal alignment deviation of text lines in adjacent areas is less than 50% of the text line height, they are determined to be continuous text areas; The criteria for determining regional integration are expressed as follows: , in, Indicates the vertical distance between adjacent regions. This indicates the area of ​​the current region.

13. The hierarchical progressive document information extraction method for ultra-high resolution images according to claim 8, characterized in that, In step S14, the selection of the cleavage line position follows the following priority rules: First priority: Select blank gap width greater than The location, among which ≥20 pixels; Second priority: Select locations that do not contain any text line bounding boxes; Third priority: Select edge locations from the edge detection results; The first level of segmentation adopts a conservative strategy, preferring to segment larger areas in order to ensure the integrity of the content. Each sub-region after segmentation contains one or more complete content units.

14. The hierarchical progressive document information extraction method for ultra-high resolution images according to claim 1, characterized in that, Step S2 includes: S21: Image Text Content Recognition For each sub-image after the first-level segmentation, extract the text content it contains for subsequent field label matching; S22: Field Label Matching and Candidate Image Index Construction Based on a predefined list of target fields, the identified text content is matched with the field labels, and an index of fields to candidate image patches is constructed.

15. The hierarchical progressive document information extraction method for ultra-high resolution images according to claim 14, characterized in that, Step S21 includes: S211: Input size judgment: The check determines whether the sub-image sizes after the first-level segmentation are within the optimal input range of the multimodal large model. The criteria are as follows: like If the result is not found, proceed directly to the text content recognition step. like Then, auxiliary secondary segmentation is needed for this sub-image; S212: Auxiliary secondary segmentation: For first-level segmented molecular images that are too large, a grid-based uniform segmentation strategy is used for further decomposition. The parameters for the auxiliary second-level segmentation are configured as follows: Grid size: 768×768 pixels to 1024×1024 pixels Overlap ratio: 20%-30%; S213: Text Content Recognition For sub-images of suitable size, a multimodal large model is used for text content recognition, and specially designed prompt words are used to guide the model to output all text information contained in the image; S214: Summary of Text Content: For images that have undergone auxiliary secondary segmentation, the text content identified by each secondary sub-block is summarized and deduplicated, and duplicate text generated by overlapping areas is removed. Finally, the summarized text content is assigned to the corresponding first-level segment sub-image, establishing a one-to-one mapping relationship between the first-level segment sub-image and its complete text content.

16. The hierarchical progressive document information extraction method for ultra-high resolution images according to claim 14, characterized in that, Step S22 includes: S221: Field vocabulary construction: Each target field is constructed with its own main label and synonyms. These labels cover various ways the field may be expressed in different documents. The field vocabulary is generated manually or through a large language model to produce synonyms for the fields to be extracted. S222: Text normalization processing: The text content and tag words of the first-level segmented molecular images are standardized, including removing redundant spaces and special characters, unifying capitalization, and removing special symbols to eliminate the impact of format differences on the matching results. S223: Field tag matching: The standardized text content is matched with the tag words of each target field. Supports both exact match and fuzzy match modes: Exact match: Checks if the text content completely contains the tag words, with the following matching conditions: ， Fuzzy matching: This method uses string similarity for more flexible matching. The similarity calculation formula is as follows: ， in, Represents the longest common subsequence. Indicates the string length, when A successful match is determined when the similarity threshold is reached. The value range is 0.7-0.9; S224: Candidate Image Index Construction: Based on the matching results, a mapping index from the target field to the first-level segmented molecular images is constructed. For each target extraction field, all first-level segmented molecular images that match the label of that field are collected, and an index structure from the field name to the candidate sub-image list is established. This index will be used for targeted processing in the extraction stage.

17. The hierarchical progressive document information extraction method for ultra-high resolution images according to claim 1, characterized in that, Step S3 includes: S31: Candidate Image Screening For each field to be extracted, the first-level segmented sub-image list corresponding to the field is obtained from the candidate image index. If no sub-image is matched for a certain field, it is marked as "not detected". Only fields with candidate images are processed for subsequent extraction. S32: Field Extraction For each field with candidate images, select an appropriate extraction strategy based on the size of the candidate images. S321: Candidate image size determination: Check the number of pixels in the first-order molecular image. Does it meet the maximum size requirement for direct input of a large multimodal model? , If the size meets the requirements, field extraction is performed directly on the sub-image. If the size does not meet the requirements, field extraction needs to be performed using the results of the second-level segmentation. S322: Extraction Strategy Selection: Direct extraction: Construct an extraction request containing the image and field names, and directly call the multimodal large model to extract field values. Extraction based on two-level segmentation: Using the second-level segmentation results generated in the multi-level segmentation module, fields are extracted from each second-level sub-block separately; S323: Extract Request Construction: Candidate image patches are combined with target field names to construct specific extraction prompts, which guide the model to extract values ​​of specified fields from the image. The model then returns the extracted values ​​of the specified fields.

18. The hierarchical progressive document information extraction method for ultra-high resolution images according to claim 17, characterized in that, In step S32, multiple extraction tasks are processed in batches in parallel, including the following steps: S321': Combine multiple field-image patches into a batch request; S322′: Utilize multithreading or asynchronous mechanisms to invoke the model in parallel; S323′: Collect and summarize the results of each extraction task.

19. The hierarchical progressive document information extraction method for ultra-high resolution images according to claim 1, characterized in that, Step S4 includes: S41: Summary of Results Collect all results returned by extraction tasks, group and summarize them by field name, and let the field set be... For fields Its candidate value set is ; S42: Voting Deduplication Mechanism When there are multiple extracted values ​​for the same field, a voting mechanism is used to select the final result: (1) Count the frequency of each value: For the field candidate values Count the number of times it appears in all the sampling results: ， in, This represents the total number of tasks extracted. For the first The result set of the extraction task For indicator functions; (2) Select the most frequent value: The value that appears most frequently is taken as the final extraction result for this field. ， (3) Confidence Calculation: The confidence level of the result is calculated based on the ratio of the highest frequency to the total number of extractions. ， in, Representation field The total number of candidate values; S43: Output Results The deduplicated extraction results are organized into a final structured output format, including field names, extracted values, confidence scores, and source image patch information.

20. A hierarchical progressive document information extraction system for ultra-high resolution images, used to perform the method according to any one of claims 1-19, characterized in that, include: The multi-level content-aware image segmentation module is used to perform the first-level segmentation, dividing the ultra-high resolution document image into several semantically complete content regions while ensuring information integrity. The semantic retrieval module is used to establish the association between image patches and target extraction fields, providing a range of candidate images for subsequent accurate extraction; The precise extraction module accurately extracts the target field values ​​for the candidate image regions located by the semantic retrieval module. The result integration and deduplication module integrates multiple candidate values ​​extracted from multiple image patches for the same field and outputs the final unique field value.

Images