Information extraction traceability method and system based on native multi-modal large model, and readable storage medium

Abstract

The invention discloses an information extraction traceability method and system based on a native multi-modal large model and a readable storage medium, and belongs to the technical field of artificial intelligence and multi-modal learning. Comprising the steps of determining a file representation form for a to-be-processed original image, performing coordinate labeling by adopting a mantissa outward rounding mode, and constructing a space labeling data set; inputting an image by using the spatial annotation data set and adopting a dynamic resolution input strategy, and training a multi-modal large language model; inputting a sample image to the trained multi-modal large language model, outputting text content and coordinate information of the sample image, and performing coordinate post-processing on the coordinate information to realize edge compensation; and outputting the text content and the post-processed coordinate information. According to the method, the problems of coordinate dislocation, information loss and the like in a traditional OCR + language model two-stage architecture can be solved, and meanwhile, the defects of precision deviation and non-uniform format of an existing MLLM in space coordinate generation are overcome.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence and multimodal learning technology, specifically relating to an information extraction and tracing method, system, and readable storage medium based on a native multimodal large model. Background Technology

[0002] With the development of artificial intelligence technology, multimodal large language models (MLLMs) have gradually become important tools for image understanding and intelligent document parsing. Traditional systems generally adopt a two-stage paradigm of "OCR first + language model post-processing": first, the OCR engine recognizes the text content and its spatial coordinates in the image, and then the recognition results are fed into the language model for semantic understanding and structure extraction.

[0003] However, this architecture has an inherent flaw—the spatial correspondence between the text fragments output by OCR and the original visual elements is easily destroyed due to recognition errors, improper segmentation, or encoding loss, resulting in the inability of subsequent information extraction results to accurately trace back to the original image position, which seriously affects the interpretability of the system and the efficiency of manual review.

[0004] To address these issues, researchers have recently begun exploring a novel technical approach: bypassing the OCR step and allowing MLLM to directly perceive and understand text and image content from the original image, achieving end-to-end information extraction. This type of "OCR-free" method avoids the accumulation of errors caused by intermediate representations, thus improving the overall robustness of the system.

[0005] In particular, some works attempt to give MLLM the ability to generate spatial coordinates. For example, models such as Shikra and mPLUG-Owl introduce coordinate regression mechanisms and support the output of bounding box coordinates in natural language format to refer to specific regions in an image.

[0006] These methods have shown initial potential in tasks such as object detection and visual question answering, marking that MLLM is evolving from pure semantic understanding to an integrated capability of "perception-understanding-localization".

[0007] However, existing methods for directly generating coordinates from MLLM still face key challenges:

[0008] (1) Since language models are essentially trained on text sequences, their ability to model continuous spatial coordinates is weak, and the generated coordinates often have slight offsets or non-standard formats.

[0009] (2) Most models can only roughly locate the target area and lack the ability to accurately capture fine-grained visual elements (such as table cells, small font fields, and densely typed text);

[0010] (3) The generated coordinate information is often not standardized and is difficult to use directly for visualization annotation or downstream automation processes, which limits its application in high-precision scenarios.

[0011] Furthermore, in practical applications such as invoice recognition, medical report analysis, and judicial evidence preservation, users not only need accurate semantic extraction results, but also the ability to precisely map each output content back to its physical location in the original image for auditing, verification, and manual checks. While existing MLLMs can output coordinates to some extent, their accuracy and stability are insufficient to support reliable traceability mechanisms.

[0012] It is worth noting that although existing research has attempted to improve the spatial awareness of MLLMs through reinforcement learning or supervised fine-tuning, and encouraged models to "generate the required output (such as bounding box coordinates, transcribed text) in the correct format," these methods have not systematically solved the problems of accuracy, consistency, and post-processing adaptation in coordinate generation. Especially when faced with complex layouts or multimodal content, the model is prone to positioning drift or formatting errors, affecting the reliability of the final results.

[0013] To address these challenges, some studies have attempted to improve the coordinate generation capabilities of MLLMs through high-quality spatial-aware fine-tuning, such as using image-text pairs with bounding box annotations for supervised training. However, these methods generally rely on general-purpose scenario data and are limited in performance on specialized documents such as financial statements and medical imaging reports—documents characterized by highly structured layouts, dense small-font text, and multimodal interleaving, placing higher demands on the model's spatial understanding and fine-grained localization capabilities. Furthermore, image input resolution processing strategies (such as fixed-resolution scaling or dynamic segmentation) significantly affect coordinate regression accuracy, but there is currently no systematic research guiding the selection of the optimal input paradigm. Therefore, constructing high-precision spatially annotated datasets for specialized domains and combining them with appropriate visual encoding strategies for end-to-end fine-tuning has become a key bottleneck in achieving reliable coordinate output. Summary of the Invention

[0014] This invention relates to a method, system, and readable storage medium for information extraction and tracing based on a native multimodal large language model. Addressing the shortcomings of existing image and text information extraction systems in terms of spatial positioning accuracy and result tracing capabilities, this invention provides a structured information extraction method based on a multimodal large language model (MLLM) for professional fields. It can take original images and natural language commands as input and directly output structured results with precise spatial coordinate annotations, achieving simultaneous generation of semantic understanding and visual positioning. It supports visual backtracking and manual verification of the extracted content. This invention solves the problems of coordinate misalignment and information loss in the traditional two-stage architecture of OCR + language model, while overcoming the accuracy deviation and inconsistent formatting defects in spatial coordinate generation of existing MLLM systems.

[0015] According to a first aspect of the technical solution of the present invention, an information extraction and tracing method based on a native multimodal large model is provided, comprising the following steps:

[0016] S1: For the original image to be processed, determine the file representation format and use the mantissa rounding method to perform coordinate annotation, and construct a spatial annotation dataset;

[0017] S2: Using the spatially labeled dataset, input images using a dynamic resolution input strategy to train a multimodal large language model;

[0018] S3: Input the sample image into the trained multimodal large language model to obtain the text content and coordinate information of the sample image; perform coordinate post-processing on the coordinate information to achieve edge compensation;

[0019] S4: Output the text content and post-processed coordinate information.

[0020] Furthermore, S1 specifically includes:

[0021] S11: For the original image to be processed, generate preliminary annotation results through a pre-annotation model;

[0022] S12: Perform preliminary verification and correction on the preliminary annotation results;

[0023] S13: Based on the preliminary annotation results after preliminary verification and correction, determine the file representation format for text entity information and table information in the original image, respectively. The file representation format includes text content and coordinate information.

[0024] S14: For the coordinate information, the coordinates are labeled by rounding the last digit outwards to form a stable coordinate space;

[0025] S15: Conduct random sampling to obtain the final spatially labeled dataset.

[0026] Furthermore, in S11, the pre-labeled model is a pipeline model.

[0027] Furthermore, in S13:

[0028] For text entity information, the JSON format {"box": [x1, y1, x2, y2], "text":"..."} is used for annotation, where box represents the bounding box coordinates of the text in the image;

[0029] For table-type content, HTML markup language with embedded spatial attributes is used, with the format as follows: ... This allows for the positioning of the entire table area.

[0030] Furthermore, in S14, the method of rounding the mantissa outwards is specifically as follows:

[0031] For the positioning coordinate information, the units digit of the x-coordinate and / or y-coordinate values ​​are rounded up to 5 or 10.

[0032] Furthermore, for high-precision scenarios (such as locating key fields in medical documents), a rounding factor of 5 is used; for layout-related tasks (such as the overall frame of a table, paragraph areas, etc.), a rounding factor of 10 is used to ensure higher coverage.

[0033] Furthermore, the multimodal large language model is Qwen2.5VL, LLava, InternVL3.5, or Ovis2.5.

[0034] Furthermore, the dynamic resolution input strategy specifically involves: adaptively dividing the image into several visual patches based on its actual size and aspect ratio, while maintaining the original pixel density, thereby generating a variable-length visual token sequence.

[0035] Furthermore, in step S3, performing coordinate post-processing on the coordinate information to achieve edge compensation specifically includes:

[0036] S31: Binarize the original image to obtain the foreground (character) region and the background region;

[0037] S32: Perform an opening operation on the foreground region to remove noise;

[0038] S33: Based on the coordinate information, traverse the four boundaries, calculate the black and white difference value corresponding to each step of outward movement for each boundary, determine the final outward expansion boundary based on the maximum difference value, and end the outward expansion if the difference values ​​are consistent.

[0039] Furthermore, in step S33, the step size is 1, meaning the system probes outward pixel by pixel to calculate the rate of change between black and white. Specifically, to speed up this calculation, the step size can be set to a larger value.

[0040] Furthermore, in step S31, the original image is binarized using the Otsu's method (OTSU).

[0041] Furthermore, the maximum inter-class variance method divides the image into foreground and background parts according to the selected threshold, calculates the inter-class variance value under the gray value, and takes the threshold / gray value corresponding to the maximum inter-class variance as the optimal threshold.

[0042] Furthermore, the formula for the Otsu's method of inter-class variance is:

[0043] Assuming a grayscale image has a background ratio of Its mean Prospect percentage Its mean The variance between classes is .

[0044] Furthermore, in S32, the opening operation includes:

[0045] Erosion: Causes the boundaries of the foreground region in an image to shrink inward;

[0046] Expansion: Expands the boundary of the foreground area outward to compensate for the width lost due to corrosion.

[0047] According to a second aspect of the present invention, an information extraction and tracing system based on a native multimodal large model is provided. The system includes a processor and a memory for storing executable instructions; wherein the processor is configured to execute the executable instructions to perform the information extraction and tracing method based on a native multimodal large model as described in any of the preceding aspects.

[0048] According to a third aspect of the technical solution of the present invention, a computer-readable storage medium is provided, wherein a computer program is stored thereon, and when the computer program is executed by a processor, it implements the information extraction and tracing method based on the native multimodal large model as described in any of the above aspects.

[0049] The beneficial effects of this invention are:

[0050] Compared with existing technologies, this invention provides an end-to-end image and text information extraction method for professional document scenarios that supports spatial coordinate tracing, and has the following significant advantages and positive technical effects:

[0051] 1. Avoid intermediate OCR steps to achieve accurate alignment of semantics and coordinates. Adopting an "OCR-free" architecture, it directly understands content from images, fundamentally avoiding the text-coordinate misalignment problem caused by traditional OCR and improving system reliability.

[0052] 2. Employing dynamic resolution input preserves high-resolution details and enhances the ability to recognize small targets. By using a dynamic resolution block segmentation strategy, the ability to capture small-sized text and compact layout elements is significantly improved while maintaining the original pixel density, solving the problem of blurred details caused by image compression in traditional methods.

[0053] 3. A unified structured output format supports the integrated expression of diverse information types. A unified and flexible JSON / HTML output format has been designed, which fully preserves the spatial location information of text entities and tables, facilitating subsequent visualization rendering, automated process integration, and manual review.

[0054] 4. An image processing-based coordinate post-processing mechanism is introduced to compensate for minor offsets and improve boundary integrity. Morphological operations (such as binarization, erosion, and dilation) are used to accurately extract the target edge contours, and the bounding box output by the model is extended to the real physical boundary, significantly improving the integrity and reliability of coordinate coverage. Attached Figure Description

[0055] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the structures shown in these drawings without creative effort.

[0056] Figure 1 This diagram illustrates an information extraction and tracing method based on a native multimodal large model.

[0057] Figure 2 A schematic diagram showing the sample and its corresponding annotation example is provided.

[0058] Figure 3 This diagram illustrates the mapping between the predicted characters and tokens.

[0059] Figure 4 This diagram illustrates how the numbers 110 and 111 are represented in the current word segmenter.

[0060] Figure 5 The diagram shows the differences in the values ​​within the labeled boxes.

[0061] Figure 6 This diagram illustrates the observed shrinkage of the model prediction box.

[0062] Figure 7 This shows the effect of rounding the last digit outwards.

[0063] Figure 8 The diagram illustrates the dataset construction process.

[0064] Figure 9 This diagram illustrates the principle of dynamic resolution input.

[0065] Figure 10 This diagram illustrates the process of extending an image along a side.

[0066] Figure 11 The diagram shows the coordinate post-processing flow.

[0067] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0068] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0069] The terms "first," "second," etc., used in this disclosure are for distinguishing similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented, for example, in orders other than those illustrated or described herein.

[0070] Furthermore, the terms “comprising” and “having”, and any variations thereof, are intended to cover non-exclusive inclusion, such that a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such process, method, product, or apparatus.

[0071] Multiple, including two or more.

[0072] And / or, it should be understood that, for the purposes of this disclosure, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone.

[0073] The technical solution of this invention first provides an information extraction and tracing method based on a native multimodal large model, including the following steps:

[0074] S1: For the original image to be processed, determine the file representation format and use the mantissa rounding method to perform coordinate annotation, and construct a spatial annotation dataset.

[0075] In a preferred embodiment, S1 specifically includes:

[0076] S11: For the original image to be processed, generate preliminary annotation results through a pre-annotation model;

[0077] S12: Perform preliminary verification and correction on the preliminary annotation results;

[0078] S13: Based on the preliminary annotation results after preliminary verification and correction, determine the file representation format for text entity information and table information in the original image, respectively. The file representation format includes text content and coordinate information.

[0079] S14: For the coordinate information, the coordinates are labeled by rounding the last digit outwards to form a stable coordinate space;

[0080] S15: Conduct random sampling to obtain the final spatially labeled dataset.

[0081] In a preferred embodiment, in step S11, the pre-labeled model is a pipeline model.

[0082] In a preferred embodiment, in S13:

[0083] For text entity information, the JSON format {"box": [x1, y1, x2, y2], "text":"..."} is used for annotation, where box represents the bounding box coordinates of the text in the image;

[0084] For table-type content, HTML markup language with embedded spatial attributes is used, with the format as follows: ... This allows for the positioning of the entire table area.

[0085] In a preferred embodiment, in S14, the method of rounding the mantissa outwards specifically is as follows:

[0086] For the positioning coordinate information, the units digit of the x-coordinate and / or y-coordinate values ​​are rounded up to 5 or 10.

[0087] In a preferred embodiment, for high-precision scenarios (such as locating key fields in medical documents), rounding is based on 5; for layout-related tasks (such as table frames, paragraph areas, etc.), rounding is based on 10 to ensure higher coverage.

[0088] S2: Using the spatially labeled dataset, input images using a dynamic resolution input strategy to train a multimodal large language model.

[0089] In a preferred embodiment, the multimodal large language model is Qwen2.5VL, LLava, InternVL3.5, or Ovis2.5.

[0090] In a preferred embodiment, the dynamic resolution input strategy specifically involves: adaptively dividing the image into several visual patches based on its actual size and aspect ratio, while maintaining the original pixel density, thereby generating a variable-length visual token sequence.

[0091] S3: Input the sample image into the trained multimodal large language model to obtain the text content and coordinate information of the sample image; perform coordinate post-processing on the coordinate information to achieve edge compensation.

[0092] In a preferred embodiment, step S3, which involves post-processing the coordinate information to achieve edge compensation, specifically includes:

[0093] S31: Binarize the original image to obtain the foreground (character) region and the background region;

[0094] S32: Perform an opening operation on the foreground region to remove noise;

[0095] S33: Based on the coordinate information, traverse the four boundaries, calculate the black and white difference value corresponding to each step of outward movement for each boundary, determine the final outward expansion boundary based on the maximum difference value, and end the outward expansion if the difference values ​​are consistent.

[0096] In a preferred embodiment, in step S33, the step size is 1, and the system probes outward pixel by pixel to calculate the rate of change between black and white. Specifically, to speed up the calculation, the step size can be set to a larger value.

[0097] In a preferred embodiment, in S31, the original image is binarized using the Otsu's method (OTSU).

[0098] In a preferred embodiment, the maximum inter-class variance method divides the image into foreground and background parts according to the selected threshold, calculates the inter-class variance value under the gray value, and takes the threshold / gray value corresponding to the maximum inter-class variance as the optimal threshold.

[0099] In a preferred embodiment, the formula for the maximum inter-class variance method is:

[0100] Assuming a grayscale image has a background ratio of Its mean Prospect percentage Its mean The variance between classes is .

[0101] In a preferred embodiment, the opening operation in step S32 includes:

[0102] Erosion: Causes the boundaries of the foreground region in an image to shrink inward;

[0103] Expansion: Expands the boundary of the foreground area outward to compensate for the width lost due to corrosion.

[0104] S4: Output the text content and post-processed coordinate information.

[0105] The present invention also provides an information extraction and tracing system based on a native multimodal large model. The system includes a processor and a memory for storing executable instructions. The processor is configured to execute the executable instructions to perform the information extraction and tracing method based on the native multimodal large model as described above.

[0106] The present invention also provides a computer-readable storage medium, wherein a computer program is stored thereon, and when the computer program is executed by a processor, it implements the information extraction and tracing method based on the native multimodal large model as described in any of the above aspects.

[0107] Example

[0108] This invention addresses the shortcomings of existing image and text information extraction systems in terms of spatial positioning accuracy and result traceability. It proposes a multimodal large language model (MLLM) structured information extraction method for professional fields, capable of directly outputting structured results with precise spatial coordinate annotations from raw images and natural language commands as input. This enables simultaneous generation of semantic understanding and visual positioning, supporting visualized backtracking and manual verification of the extracted content. It solves problems such as coordinate misalignment and information loss in the traditional two-stage architecture of OCR + language model, while overcoming the accuracy deviations and inconsistent formats in spatial coordinate generation of existing MLLM systems. Figure 1 As shown, in the technical solution of the present invention:

[0109] Dataset Optimization: Methods for Constructing High-Precision Spatial Labeling Datasets for Professional Fields

[0110] This aspect includes innovation in labeled data content and optimization of the dataset collection process. Combining these two methods ensures the accuracy of coordinate localization in the labeled data, minimizing potential model accuracy anomalies caused by coordinate discrepancies in manually labeled data.

[0111] Model optimization: Multi-scale fine-tuning strategy supporting dynamic resolution input

[0112] This approach utilizes a special model architecture to extract features from images of any scale while effectively preserving the original image features.

[0113] Post-processing optimization: Coordinate post-processing and edge compensation mechanism based on visual enhancement

[0114] This aspect utilizes a series of specific image operations to optimize the risk of coordinate offset caused by model prediction, effectively ensuring the correspondence between the final coordinates and the elements in the image.

[0115] The following is a detailed description of each part.

[0116] Methods for constructing high-precision spatial annotation datasets for professional fields

[0117] To construct a large-scale image-text pair dataset for documents with complex layouts, we first present the representation of the annotation file with spatial information for each image:

[0118] For text entity information (such as "patient name", "amount", "date", etc.), the JSON format {"box": [x1, y1,x2, y2], "text": "..."} is used for annotation, where box represents the bounding box coordinates of the text in the image;

[0119] For table-type content, HTML markup language with embedded space attributes is used, in the format of: ... To achieve positioning of the entire table area, such as Figure 2 As shown. Among them, Figure 2 This shows a specific sample (above) and its corresponding JSON annotations (below).

[0120] After confirming the annotation file format, two optimizations were made based on the current annotation process: First, the coordinate annotation content was optimized by using the "rounding out" method to modify the proportion of each number, which effectively reduced the training difficulty of the model without significantly affecting the inference results; Second, a complete data production process was designed to ensure that the final rounded annotation file would not have significant deviations when rendered to the image, thus avoiding poor model performance caused by training data.

[0121] Coordinate labeling optimization: Rounding off the last digit.

[0122] Regarding coordinate labeling, we start with model analysis: the character set and number set corresponding to the token predicted by the model are inconsistent. The numbers in the model may be 0-9, or they may be specific numerical values. This is generally related to the frequency of numerical values ​​in the dataset when the tokenizer is built, such as... Figure 3 As shown.

[0123] This leads to a problem: semantically similar numbers can have vastly different encodings. Figure 4 For example, when the tokenizer encodes 110 and 111, since numbers are also represented using the character set in the dictionary, the situation shown in the diagram might occur: similar coordinates generate different numbers of tokens. As can be seen from the diagram, representing 110 requires 2 tokens, but representing 111 only requires 1 token. Since our coordinates can take any value, this undoubtedly increases the cost of model learning and inference when the model needs to be accurate to single digits.

[0124] On the other hand, through observation of the trained model, technicians discovered that the model is prone to shrinkage when generating coordinates. Specifically, refer to... Figure 6 This means the predicted bounding box is slightly smaller than the actual element area, often manifesting as truncated or incomplete text or table edges. Based on the previous analysis of the tokenizer, this problem mainly stems from the following two reasons:

[0125] 1) Differences in numerical discretization: During the construction of the tokenizer, similar numbers (such as 110 and 111) may be split into different numbers of tokens, making it difficult for the model to align coordinates during prediction and increasing the learning difficulty.

[0126] 2) Fine-grained prediction is costly: When the model needs to be accurate to single digits, the prediction space is large, which easily introduces fluctuations and errors. At the same time, for similar samples, the bounding boxes obtained by different processing may also have numerical differences, ultimately resulting in offset or shrinkage (hereinafter). Figure 5For example, if there are three samples [10, 11, 100], and the corresponding tokens are [10, 11, [10, 0]], the model finds that 10 and 11 are correct when learning the first two samples. Assuming the sample ratio corresponds to the probability generated by the model, and the model ultimately predicts the third sample, when predicting the first token for the coordinates, it might assume that predicting coordinate 10 is 50% and predicting 11 is 50%, so the model still has a certain probability of choosing 11. However, in reality, there is a huge difference of 10 pixels between the final prediction results 100 and 110. Therefore, it can be seen that small differences in annotation, a wide prediction space, and the combined effects of numerical tokenization can amplify small differences in the model's coordinate predictions.

[0127] To alleviate the aforementioned problems, this invention proposes a method for rounding the last digit outwards. The core idea is to uniformly round the units digit of the predicted coordinates according to a fixed rule, mapping all coordinates to a more stable and discrete numerical space, thereby reducing learning complexity and enhancing prediction robustness. Furthermore, rounding down may result in the bounding box not covering the entire element content, while rounding outwards can offset some of the bias in model predictions; the model predictions tend to leave some margin, thus canceling out prediction errors in the tens place.

[0128] like Figure 7 The specific method is as follows:

[0129] Rounding rule: Round the units digit of the x-coordinate and / or y-coordinate values ​​outwards, using 5 or 10 as the baseline. For example: If a predicted coordinate is (233, 233), if it is the top left corner of the image, when expanding outwards, the x-value can be moved to the left to become (230, 233), the y-value can be moved upwards to become (233, 235), or the x-value can be moved to the left and the y-value can be moved upwards to become (230, 235); if it is the bottom left corner of the image, when expanding outwards, the x-value can be moved to the left to become (230, 233), the y-value can be moved downwards to become (233, 230), or the x-value can be moved to the left and the y-value can be moved downwards to become (230, 230); if it is the top right corner of the image, when expanding outwards, the x-value can be moved to the right to become (235, 230 ... (233), or the y-value can be moved upwards to (233, 235), or the x-value can be moved to the right while the y-value moves upwards to (235, 235). If it is the bottom right corner of the image, when expanding outwards, the x-value can be moved to the right to (235, 233), or the y-value can be moved downwards to (233, 230), or the x-value can be moved to the right while the y-value moves downwards to (235, 230). Similarly, if the predicted coordinate is 487, it will be rounded to 490.

[0130] Adaptive adjustments: For high-precision scenarios (such as locating key fields in medical documents), rounding based on 5 can be used; for layout-related tasks (such as table frames, paragraph areas, etc.), rounding based on 10 can be used to ensure higher coverage.

[0131] This method eliminates the need for the model to repeatedly choose between adjacent numbers during prediction; instead, it directly selects from a "normalized" coordinate space. This not only reduces the types and number of coordinate tokens but also improves the robustness of the bounding box in covering the target area, effectively mitigating the problem of indentation in the prediction results. Experimental results show that using the rounding-out method improves both the recall rate of document elements and the overall layout accuracy.

[0132] Data production process

[0133] This invention uses existing small model pipelines for pre-annotation, generates preliminary annotation results through initial model screening, then performs manual verification and correction, and then uses a script to round the coordinates of all structured data outwards before sampling verification.

[0134] Ultimately, all labeled data is generated through a collaborative workflow of "pre-labeling model initial screening, high-precision manual verification and correction, and post-processing coordinate correction," significantly improving labeling consistency and accuracy while ensuring labeling efficiency. Figure 8 As shown.

[0135] Multi-scale fine-tuning strategy supporting dynamic resolution input

[0136] This invention proposes a multi-scale fine-tuning method for general visual multimodal models (not limited to Qwen2.5VL, LLava, InternVL3.5, Ovis2.5, and other related models), primarily focusing on optimizations in the Vit (VisionTransformer) part. This method moves away from fixed-resolution scaling during the image input stage, employing a dynamic resolution input strategy. It adaptively segments the image into several visual patches based on its actual size and aspect ratio, while maintaining the original pixel density, thereby generating a variable-length visual token sequence. Figure 9 As shown.

[0137] In the past, most models pre-padding and resizing images to a specific scale before segmenting and processing them, and finally sampling to obtain multiple fixed-length vectors as a feature vector group for the image. During the initial padding and scaling operations, some content features of the image may be distorted. For example, if an 84*28 image is stretched horizontally to 448*28, the original text features will be abnormal, and this implicit error may be propagated incorrectly. However, if the results of segmentation based on the image size are directly used to represent the image, the distortion problem caused by scaling can be completely avoided without resizing. Furthermore, the difference in the number of image features in subsequent stages allows the LLM stage to focus more on the main image features.

[0138] Compared to methods that rely on fixed resolution scaling or simply normalized coordinate mapping, the dynamic resolution scheme of this invention has advantages in the following aspects:

[0139] 1. Fine-grained feature preservation: Avoids font blurring or loss of small target information caused by uniform scaling, and improves sensitivity to small text areas and complex layout structures.

[0140] 2. Input Adaptability: Documents of different sizes and proportions can be encoded into token sequences without destroying geometric information, enhancing the model's ability to generalize to cross-scene inputs.

[0141] Coordinate post-processing and edge compensation mechanism based on visual enhancement

[0142] To address the issue of slight offsets or incomplete coverage of target edges in the model's output coordinates, this invention introduces a lightweight post-processing workflow to further refine coordinate accuracy. The complete workflow is as follows: Figure 11 As shown, the specific steps include the following three parts:

[0143] 1. Binarize the original image.

[0144] The OTSU algorithm is typically used to automatically determine the binarization threshold. The basic idea is to divide the image into two parts—target and background—based on a selected threshold, calculate the inter-class variance at that grayscale value, and take the threshold / grayscale value corresponding to the maximum inter-class variance as the optimal threshold. For example, for a grayscale image where all values ​​are between 0 and 100, with most concentrated between 10 and 20 and 40 and 60, the threshold obtained using the OTSU algorithm will likely be around 25 or 35. This method of automatically determining the threshold based on different grayscale images is much more effective than manually setting it to 127.

[0145] The specific calculation formula for the OTSU algorithm is as follows:

[0146] Assuming a grayscale image has a background ratio of Its mean Prospect percentage Its mean Let the variance within the class be denoted as . Then the variance between classes is Its equivalent (refer to the following formula) is: .

[0147] Equivalent transformation formula of the inter-class variance formula

[0148]

[0149] Next, iterate through the threshold values ​​from 0 to 255, calculate the corresponding inter-class variance values, and then take the maximum value.

[0150] 2. Apply morphological operations: Perform opening operation to remove noise.

[0151] The opening operation involves erosion followed by dilation. This step is for further refinement and noise reduction.

[0152] Noise reduction principle:

[0153] Erosion: Any isolated noise smaller than the structuring element (such as scan blemishes or tiny burrs) will be completely eroded away and disappear from the image.

[0154] Erosion operation. The erosion operation causes the boundaries of the foreground region (characters) in an image to shrink inward, thus "thinning" the characters. When there is only a slight connection (adhesion) between two characters, by shrinking the boundaries, these slight connections will be completely "eroded" away, thus achieving physical separation of the characters.

[0155] Inflation: Since the noise has been removed, this inflation will only make the remaining main characters slightly larger, without restoring the noise that has disappeared.

[0156] Dilation operation. After erosion separates the adhered parts, dilation is used to "bold" the characters back to near their original size and shape. The dilation operation expands the boundaries of the foreground area (characters) outward to compensate for the width lost due to erosion. At this point, the main body of the characters is restored, but the connections (adhesions) between them, because they were completely severed in the first step, will not be restored.

[0157] 3. Accurately locate object edges based on the processed binary image, and extend the initial output box coordinates of the model to the actual physical boundary.

[0158] By traversing the four boundaries, the black-and-white difference value corresponding to each step of outward movement is calculated for each boundary. The final outward expansion boundary is determined based on the maximum difference. The outward expansion ends when the differences are almost identical. Figure 10What is shown is the calculation process that expands downwards and outwards.

[0159] This method can significantly improve the spatial integrity and visual fit of bounding boxes without increasing the burden of model inference, thus further ensuring the reliability of the tracing results.

[0160] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0161] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0162] Through the above description of the embodiments, those skilled in the art can clearly understand that the above implementation methods can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0163] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other modifications under the guidance of the present invention without departing from the spirit and scope of the claims. All of these modifications are within the protection scope of the present invention.

Claims

1. A method for information extraction and tracing based on a native multimodal large model, characterized in that, Includes the following steps: S1: For the original image to be processed, determine the file representation format and use the mantissa rounding method to perform coordinate annotation, and construct a spatial annotation dataset; S2: Using the spatially labeled dataset, input images using a dynamic resolution input strategy to train a multimodal large language model; S3: Input the sample image into the trained multimodal large language model to obtain the text content and coordinate information of the sample image; perform coordinate post-processing on the coordinate information to achieve edge compensation; S4: Output the text content and post-processed coordinate information.

2. The information extraction and tracing method based on a native multimodal large model according to claim 1, characterized in that, S1 specifically includes: S11: For the original image to be processed, generate preliminary annotation results through a pre-annotation model; S12: Perform preliminary verification and correction on the preliminary annotation results; S13: Based on the preliminary annotation results after preliminary verification and correction, determine the file representation format for text entity information and table information in the original image, respectively. The file representation format includes text content and coordinate information. S14: For the coordinate information, the coordinates are labeled by rounding the last digit outwards to form a stable coordinate space; S15: Conduct random sampling to obtain the final spatially labeled dataset.

3. The information extraction and tracing method based on a native multimodal large model according to claim 2, characterized in that, In S11, the pre-labeled model is a pipeline model.

4. The information extraction and tracing method based on a native multimodal large model according to claim 2, characterized in that, In S13: For text entity information, the JSON format {"box": [x1, y1, x2, y2], "text":"..."} is used for annotation, where box represents the bounding box coordinates of the text in the image; For table-type content, HTML markup language with embedded spatial attributes is used, with the format as follows:<tablebox="[x1,y1,x2,y2]"> ...to achieve positioning of the entire table area.

5. The information extraction and tracing method based on a native multimodal large model according to claim 2, characterized in that, In S14, the method of rounding the mantissa outwards is specifically as follows: For the positioning coordinate information, the units digit of the x-coordinate and / or y-coordinate coordinate values ​​are rounded outwards to a base of 5 or 10.

6. The information extraction and tracing method based on a native multimodal large model according to claim 1, characterized in that, In S2, the multimodal large language model is Qwen2.5VL, LLava, InternVL3.5, or Ovis2.

5.

7. The information extraction and tracing method based on a native multimodal large model according to claim 1, characterized in that, In step S2, the dynamic resolution input strategy specifically involves: adaptively dividing the image into several visual patches based on its actual size and aspect ratio, while maintaining the original pixel density, thereby generating a variable-length visual token sequence.

8. The information extraction and tracing method based on a native multimodal large model according to claim 1, characterized in that, In step S3, performing coordinate post-processing on the coordinate information to achieve edge compensation specifically includes: S31: Binarize the original image to obtain the foreground and background regions; S32: Perform an opening operation on the foreground region to remove noise; S33: Based on the coordinate information, traverse the four boundaries, calculate the black and white difference value corresponding to each step of outward movement for each boundary, determine the final outward expansion boundary based on the maximum difference value, and end the outward expansion if the difference values ​​are consistent.

9. The information extraction and tracing method based on a native multimodal large model according to claim 8, characterized in that, In S33, the step size of each outward movement is one pixel.

10. The information extraction and tracing method based on a native multimodal large model according to claim 8, characterized in that, In step S31, the original image is binarized using the maximum inter-class variance method.

11. The information extraction and tracing method based on a native multimodal large model according to claim 10, characterized in that, The maximum inter-class variance method divides the image into foreground and background parts based on the selected threshold, calculates the inter-class variance value at the gray value, and takes the threshold / gray value corresponding to the maximum inter-class variance as the optimal threshold.

12. The information extraction and tracing method based on a native multimodal large model according to claim 10, characterized in that, The formula for the Otsu's method of inter-class variance is: Assuming a grayscale image has a background ratio of Its mean Prospect percentage Its mean The variance between classes is .

13. The information extraction and tracing method based on a native multimodal large model according to claim 8, characterized in that, In S32, the opening operation includes: Erosion: Causes the boundaries of the foreground region in an image to shrink inward; Expansion: Expands the boundary of the foreground area outward to compensate for the width lost due to corrosion.

14. An information extraction and tracing system based on a native multimodal large model, the system comprising: A processor and a memory for storing executable instructions; wherein the processor is configured to execute the executable instructions to perform the information extraction and tracing method based on a native multimodal large model as described in any one of claims 1 to 13.

15. A computer-readable storage medium, wherein, It stores a computer program, which, when executed by a processor, implements the information extraction and tracing method based on a native multimodal large model as described in any one of claims 1 to 13.

Images