An information extraction and tracing method and system based on cooperation of an OCR model and a large language model, and a readable storage medium
By combining OCR and a large language model, accurate traceability of text and image information is achieved, solving the problem of disconnect between structured information output and image coordinates in existing technologies. This improves the verifiability and computational efficiency of the extraction results, adapts to various document formats, and supports private deployment.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- BEIJING YIDAO BOSHI TECH
- Filing Date
- 2026-01-06
- Publication Date
- 2026-05-29
AI Technical Summary
In existing technologies, there is a problem that the structured output of image and text information extraction is disconnected from the coordinates of the original image. In particular, when using large language models (LLM), there is a lack of location information, which makes it difficult to trace the source and verify. Moreover, end-to-end multimodal models consume high computational resources in complex scenarios and have poor applicability.
We employ an information extraction method based on the collaboration of OCR and large language models. Through a modular architecture, OCR is responsible for text and coordinate extraction, while LLM is responsible for semantic parsing. Combined with a multi-level text alignment algorithm, we achieve precise source tracing at the field level.
It enables precise tracing of the output results of large language models back to the original image coordinates, improves the verifiability and robustness of the extraction results, reduces computational costs, adapts to various complex document formats, and supports private deployment and domestic adaptation.
Smart Images
Figure CN122116393A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of computer vision and natural language processing technology, specifically relating to an information extraction and tracing method, system, and readable storage medium based on the collaboration of OCR model and large language model. Background Technology
[0002] In existing technologies, image and text information extraction typically relies on OCR technology to first obtain the text content and corresponding coordinates, and then uses manual rules or machine learning models to locate and recognize fields. In such pipelines, there is often a problem of disconnect between the structured output and the original image coordinates: even when using a Large Language Model (LLM) to extract fields from the OCR text, the LLM output is only the text result, without corresponding image location information, making it difficult to trace and verify the source.
[0003] In addition, although end-to-end multimodal document understanding models (such as LayoutLM, StrucTexT, LayoutXLM, etc.) can directly process images and output structured information, they lack source information and usually require large model computing resources, dedicated fine-tuning data and expensive hardware, making them inconvenient to use in private deployments and heterogeneous scenarios.
[0004] More importantly, these models often perform poorly in special scenarios such as complex formats, non-standard layouts, or handwritten content. Therefore, the industry urgently needs a new method that can both utilize the semantic extraction capabilities of LLM and accurately trace back to the original image text region to solve the problem of "separation of LLM output and image coordinate information". Summary of the Invention
[0005] To address the aforementioned problems, this invention proposes an information extraction and tracing method based on the collaboration of an OCR model and a large language model, enabling precise tracing of LLM structured extraction results back to the original image coordinates. Specifically, this invention aims to overcome the lack of positional information in the output fields of large models in existing LLM extraction pipelines, improving the verifiability and traceability of extraction results. Furthermore, compared to end-to-end multimodal large models, this invention adopts a modular architecture: a small OCR model handles efficient text and coordinate extraction, while the LLM model handles semantic parsing; the two can be flexibly replaced with a lighter computational burden. This invention designs a multi-layered text alignment algorithm to perform fine-grained matching and positioning of LLM output fields. Whether it's complete field matching or partial substring matching, it can locate the corresponding region in the original image, achieving "precise tracing" at the field level.
[0006] According to a first aspect of the technical solution of the present invention, an information extraction and tracing method based on the collaboration of OCR model and large language model is provided, comprising the following steps: S1: Input the original image, and output the text content and corresponding coordinate information through the OCR model; S2: Concatenate the text content into a string and input it into the large language model to construct prompts for field extraction; S3: Match the fields extracted by the large language model with the text content output by the OCR model. If there is only one match, output it directly. If there are multiple matches, perform disambiguation based on the heuristic scoring mechanism of context semantics and spatial layout to determine the optimal matching result. If there are no matches, use the sliding window weighted edit distance algorithm for fault-tolerant matching. S4: Output the final matching result, including the fields and their corresponding coordinate information.
[0007] Furthermore, in step S2, when concatenating the text content into a string, the text is concatenated line by line according to the reading order, with newline characters separating the lines.
[0008] Furthermore, in S2, the extracted fields are output in a JSON structure.
[0009] Furthermore, S3 specifically includes: S31: The fields extracted by the large language model and the text content output by the OCR model are standardized and preprocessed; S32: For the standardized and preprocessed field, perform an exact substring search in the text content output by the OCR model. If there is only one match, proceed to S35; if there are multiple matches, proceed to S33; if there are no matches, proceed to S34. S33: Based on the heuristic scoring mechanism of contextual semantics and spatial layout, multiple matching items are scored separately for disambiguation, and the highest score is selected as the optimal matching result. Then, S35 is executed. S34: Use the sliding window weighted edit distance algorithm for fault-tolerant matching. Select the text content output by the OCR model with a similarity higher than the threshold as the optimal matching result, and then execute S35. S35: Determine the logical position of the field in the text content output by the OCR model, and convert it into physical coordinates of the image; The order of steps 33 and 34 can be adjusted.
[0010] Furthermore, in S31, the standardization and preprocessing specifically include unifying font case and removing redundant whitespace symbols.
[0011] Furthermore, in step S32, the field extracted by the large language model is Q, and the text content output by the OCR model is... The matching objective function is defined as finding the function that satisfies... The row index i and the character start index j, This is the length of field Q.
[0012] Furthermore, S33 specifically includes: Suppose that K candidate matches are found in the text output by the OCR model, denoted as set K. ,in This represents the row index of the k-th candidate match. This represents the starting character index of the k-th candidate match; The score for each candidate is calculated using the following formula. : Explanation of formula variables: : The total confidence score of the k-th candidate matching position; Weight hyperparameters are used to adjust the weighting of context, layout, and text matching. Context similarity function, in candidate rows neighborhood Search for the "key" corresponding to this field. If it exists, the score for that item is close to 1; otherwise, it is close to 0. : Layout prior probability function, which defines the probability distribution of the position of a field based on the document type; if the position conforms to the prior distribution, a higher score is assigned; Edit the distance penalty item; The text content of the k-th candidate match; This is the length of field Q.
[0013] Furthermore, take .
[0014] Furthermore, in step S34, the field extracted by the large language model is Q, and each line of OCR text in the text content output by the OCR model is... Construct a length of The sliding window, in which Initialized to the length of field Q ; During the sliding process, the similarity Sim between the substring sub and field Q within the window is calculated: The adaptive threshold determination logic is as follows: Explanation of formula variables: : Levinstein distance function, calculates the minimum number of single-character edit operations required to convert the substring sub into the field Q; : Normalization factor, ensuring similarity is within the interval [0, 1]; where, is the length of the substring sub; High confidence threshold: If the similarity is higher than this value, it is directly considered a successful match; The low confidence threshold, if it is in the middle range, needs to be further confirmed by combining the context score from the previous step to prevent false matching.
[0015] Furthermore, the aforementioned The value is 0.9, the The value is 0.7.
[0016] Furthermore, in step S34, if there are no matches with similarity higher than the threshold, the anchor point extension algorithm is used to perform cross-line splicing and anchor point positioning, thereby selecting matches with similarity higher than the threshold as the optimal matching result, and then step S35 is executed.
[0017] Furthermore, the anchor point extension algorithm specifically includes: Cross-line text aggregation: Detecting the length of field Q extracted from a large language model ,like If the number of characters exceeds the average number of characters in a single line of OCR, then construct a multi-line concatenated string. : in, For a multi-line text block concatenated starting from line k, the string is composed of... It is pieced together; This represents a string concatenation operation. As a line separator; in Perform a fuzzy search to determine the starting row of field Q. and end line ; Anchor point localization and coordinate fusion: If direct splicing and matching are difficult, an "anchor point" strategy is adopted. The longest substring with the highest distinguishability in Q is selected as the anchor point A. First, A is located in the text content output by the OCR model, and then the search is expanded to the left and right to find the remaining characters. Finally, the overall physical coordinates of the field are fused. It is calculated from the union of the bounding boxes of all covering characters: Explanation of formula variables: : Represents the set of all matched characters, which spans multiple lines of OCR text; The coordinates of the top-left corner of the c-th character in the set; The coordinates of the bottom right corner of the c-th character in the set; The final calculated rectangle is the smallest bounding rectangle that can enclose the multi-line text.
[0018] Furthermore, S35 specifically involves: assuming the character-level coordinate sequence of the i-th row provided by the OCR engine is... ,in A bounding box for a single character. Then the overall coordinates of field Q. Calculate the union of all character bounding boxes within the interval.
[0019] Furthermore, in S4, the output format is a list of JSON objects, each object containing field_name, field_value, and bbox (boundary box coordinates).
[0020] According to a second aspect of the present invention, an information extraction and tracing system based on the collaboration of an OCR model and a large language model is provided. The system includes: a processor and a memory for storing executable instructions; wherein the processor is configured to execute the executable instructions to perform the information extraction and tracing method based on a native multimodal large model as described in any of the preceding aspects.
[0021] According to a third aspect of the present invention, a computer-readable storage medium is provided, wherein a computer program is stored thereon, and when the computer program is executed by a processor, it implements the information extraction and tracing method based on the collaboration of OCR model and large language model as described in any of the above aspects.
[0022] The beneficial effects of this invention are: 1. This invention achieves precise bidirectional backtracking based on semantics and coordinates, solving the localization problem caused by the "illusion" of Large Language Models (LLMs). Existing Large Language Models (LLMs) often lack physical coordinates in their generated results and are prone to producing characters that are not present in the original image (illusion). This invention establishes fine-grained character-by-character alignment at the semantic level of the text by constructing a hierarchical backtracking mechanism of "precise matching - heuristic scoring - fuzzy search". In particular, the introduced context-based heuristic scoring algorithm uses neighborhood keywords and layout prior knowledge for disambiguation, ensuring that the target field can be accurately located even when multiple identical values (such as multiple dates or amounts) appear in the document. This achieves bidirectional verifiability between structured data and the original image region, significantly improving the credibility of the data.
[0023] 2. Highly robust OCR noise tolerance. Addressing common OCR issues such as character misidentification (e.g., confusion between 0 and O), missed identification, or line breaks, this invention abandons traditional rigid regular expression matching and adopts a fuzzy matching strategy based on weighted edit distance. Combined with an adaptive threshold determination mechanism, the system can tolerate slight differences between the LLM output and the original OCR text (e.g., punctuation, capitalization, or individual typos). Experiments show that even when OCR accuracy is limited (e.g., seal obscuring, blurred text), this method can still accurately locate the image region corresponding to the field through semantic similarity calculation, significantly enhancing the system's usability in complex real-world scenarios.
[0024] 3. Overcoming the bottleneck of locating long text and multi-line fields. For long text fields such as company names and long notes that are easily truncated into multiple lines by OCR, the "anchor point extension and multi-line aggregation" algorithm proposed in this invention effectively solves the problem of single-line matching failure. By locating highly unique "anchor point" substrings and dynamically extending them in all directions, the system can automatically calculate and merge the bounding boxes of multi-line text to generate a complete minimum bounding rectangle. This design significantly improves the completeness of information extraction from unstructured long documents (such as contract terms and long descriptions).
[0025] 4. Decoupled architecture separating large and small models reduces computational costs and deployment barriers. Compared to end-to-end multimodal large models such as LayoutLM and GPT-4v, this invention adopts a decoupled architecture of "small model OCR + general-purpose LLM". OCR preprocessing can be completed by a lightweight CPU model, and the LLM only needs to process plain text data, without consuming the computational power of an expensive visual encoder. This architecture not only significantly reduces inference costs and memory usage, but also supports modular replacement (such as replacing with a more powerful OCR engine or a cheaper LLM), greatly facilitating private deployment and domestic adaptation.
[0026] 5. Strong versatility and zero-sample / few-sample capability without templates. This invention leverages the powerful semantic understanding capabilities of LLM, completely eliminating the reliance on "document templates" or "regular expressions" found in traditional methods. Users only need to provide simple field definitions or a few examples (Few-shots), and the system can adaptively process various documents with vastly different formats, such as invoices, contracts, medical documents, and customs declarations. This "input-and-use" feature significantly reduces the adaptation cost to new scenarios and the workload of manually customizing rules.
[0027] 6. Improve the efficiency and compliance of downstream applications. Thanks to the precise coordinate information included in the output results, this invention can directly empower automated auditing, financial reimbursement verification, and manual review systems. On the user interface, the system can highlight the position of extracted fields in the original image, assisting manual comparison and confirmation. This not only improves the automation level of business processes but also meets the stringent compliance requirements of industries such as finance and law regarding the "traceability and auditability" of data sources.
[0028] In summary, this invention successfully solves the industry pain point of "position loss" in large model information extraction by integrating semantic alignment algorithms and multi-layer geometric calculation strategies. While ensuring high accuracy, it also takes into account system robustness, computational economy and scenario versatility, and has extremely high practical value. Attached Figure Description
[0029] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the structures shown in these drawings without creative effort.
[0030] Figure 1 The flowchart of an information extraction and tracing system based on the collaboration of OCR model and large language model is shown.
[0031] Figure 2 The flowchart of the text matching and coordinate tracing module in the information extraction and tracing system based on the collaboration of OCR model and large language model is shown.
[0032] Figure 3 An extraction effect diagram is shown according to an embodiment of the present invention.
[0033] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation
[0034] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.
[0035] The terms "first," "second," etc., used in this disclosure are for distinguishing similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented, for example, in orders other than those illustrated or described herein.
[0036] Furthermore, the terms “comprising” and “having”, and any variations thereof, are intended to cover non-exclusive inclusion, such that a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such process, method, product, or apparatus.
[0037] Multiple, including two or more.
[0038] And / or, it should be understood that, for the purposes of this disclosure, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone.
[0039] Example like Figure 1 As shown, the information extraction and precise coordinate tracing method described in this embodiment mainly includes the following steps: Step 1: OCR Recognition. A traditional OCR mini-model is used to perform OCR recognition on the input image, extracting the content and coordinate information of all visible text lines in the image. The OCR module records the text content and coordinate information of each segment of text in a line-by-line manner.
[0040] Step 2: Structured Field Extraction from the Large Model. The entire text content extracted by OCR (which can be concatenated line by line in reading order, separated by newlines) is fed as input to the large language model. Using pre-designed prompts and a few examples, the large model is guided to output structured field results (e.g., in JSON format), including the names of the various entities or fields to be extracted and their corresponding text values. This step leverages the semantic understanding capabilities of LLM to perform deep parsing of the text content.
[0041] Step 3: Text Matching and Coordinate Tracing. For each extracted field in the output of the large model, precise matching and error-tolerant alignment are performed in the OCR result text, tracing its origin back to the image coordinates. This step is the focus of this invention.
[0042] Figure 1 The diagram shows the system flow of this method: First, the OCR module extracts the text and coordinates, then the LLM module extracts the fields, then the text matching and coordinate backtracking module accurately maps the extraction results to the original image position, and finally integrates and outputs the information extraction field results with coordinates.
[0043] The implementation process of the present invention will be described in detail below with reference to specific steps.
[0044] OCR recognition The input image is preprocessed, and a small OCR model is used to recognize all the text in the image. The OCR output includes several lines of text, each containing plain text content and corresponding coordinate information.
[0045] Large model structured field extraction After sorting the OCR results in reading order, concatenate each line of text into a single large string (newline characters can be added at the end of each line to separate them), constructing the LLM input. Then, use the large model to extract fields, for example, constructing a prompt like: "Extract invoice number, date, total amount, etc. from the following text. Please only return results in JSON format." Incorporate domain examples as needed. The output of the large model is generally in JSON structure, such as: {"invoice_no":"ABC123","date":"2025-09-01","total_amount":"1000.00"}. At this point, the text content of each field is obtained, but the coordinates are not yet included.
[0046] Text matching and coordinate tracing The focus of this invention is this step, which maps each LLM output field value to the OCR result using a precise and robust algorithm to obtain the corresponding image coordinates. This process employs a coarse-to-fine layered matching strategy, combining precise matching, heuristic scoring, and fuzzy error-tolerant algorithms. The specific flow is as follows (see flowchart for this module). Figure 2 ): Text Standardization and Preprocessing: To reduce matching difficulty, the system first performs unified standardization processing on the OCR recognition result set and the LLM output field set. This processing includes case neutralization and removal of redundant whitespace characters.
[0047] Exact Matching and Candidate Set Construction: For any text field Q to be traced, the system first compares it with the OCR text line list. Perform an exact substring search. Define the matching objective function as finding substrings that satisfy... The row index i and the character start index j.
[0048] If the search is successful, add all matching terms to the candidate set. At this point, if the set If there is only one match, it will be directly identified as the final position; if there are multiple matches (e.g., the same amount or date appears multiple times in the document), the following steps will be performed for disambiguation.
[0049] Multiple Candidate Location and Heuristic Scoring: When the field value Q extracted by LLM appears multiple times in the OCR text (e.g., in an invoice, 200.00 could be either "unit price" or "amount"), the system cannot determine the unique location solely through string matching. In this case, this step introduces a heuristic scoring mechanism based on contextual semantics and spatial layout.
[0050] Suppose that K candidate matching positions are found in the OCR text, denoted as set K. ,in Indicates row index, Indicates the starting index of the character. The system calculates the score for each candidate using the following formula. : Explanation of formula variables: : The total confidence score of the k-th candidate matching position.
[0051] : Weight hyperparameter, used to adjust the influence ratio of context, layout, and text matching. In this embodiment, we take... .
[0052] Context similarity function. The system will consider candidate rows. neighborhood Search for the "key" corresponding to the field in the preceding 5 characters or the previous line of text.
[0053] Calculation logic: If the field name is "Total Amount" and keywords such as "Total" or "Amount in Total" are detected to the left of the candidate position, the score of this item is close to 1; otherwise, it is close to 0.
[0054] Layout prior probability function. Defines the probability distribution of field occurrences based on document type (e.g., invoice).
[0055] Calculation logic: For example, "invoice code" is usually located at the top of the page ( (Smaller score), "Notes" is located at the bottom. If the position matches the prior distribution, a higher score is assigned.
[0056] : Edit distance penalty. Used for fine-tuning, prioritizing matches with exactly identical characters and penalizing candidates with minor differences.
[0057] [Application Scenario Example - Invoice]: LLM extracts a "Total Amount" of 1000.00. In the OCR result, 1000.00 appears in two places: Position A: The preceding text is "Unit Price:".
[0058] Position B: The preceding text is "Total Price Including Tax:". The system calculates... Position A matched "unit price," which did not match the target field "total amount," resulting in a low score; Position B matched "total," which was strongly correlated with the target field, resulting in a high score. Ultimately, the system automatically selected Position B.
[0059] Fuzzy matching based on edit distance: If no exact match is found in all rows, it indicates that there may be a recognition error in OCR (such as misidentifying 0 as O) or slight rewriting in LLM output. This step uses a sliding window weighted edit distance algorithm for fault-tolerant matching.
[0060] Specifically, for each line of OCR text The system constructs a length of The sliding window, in which Initialized to the length of field Q During the sliding process, the similarity Sim between the substring sub and field Q within the window is calculated: To further improve robustness, this invention introduces adaptive threshold determination logic: Explanation of formula variables: : Levenstein distance function, calculates the minimum number of single-character editing operations (insertion, deletion, replacement) required to convert the substring sub into the field Q.
[0061] : Normalization factor, ensuring similarity is within the range [0, 1].
[0062] High confidence threshold (e.g., 0.9). If the similarity is higher than this value, it is considered a successful match.
[0063] Low confidence threshold (e.g., 0.7). If it falls within the middle range, further confirmation is needed based on the context score from the previous step to prevent false matches.
[0064]
Application Scenario Example - Invoice
[0065] Exact match: failed.
[0066] Fuzzy matching: calculation and The edit distance is 1. Similarity .
[0067] determination: And the prefix of that line is If the context verification passes, the match is determined to be successful, and the coordinates are traced back.
[0068] Line concatenation and anchor point positioning: When field content is too long, causing line breaks (such as long company names or remarks), or when OCR breaks a line of text into multiple blocks, single-line matching will fail. This invention uses an anchor point extension algorithm to solve this problem.
[0069] Step A: Cross-line text aggregation. First, check the length of the LLM output field Q. If... If the number of characters is significantly greater than the average number of characters in a single line of OCR, then a multi-line concatenated string is constructed. : in This represents a string concatenation operation. This is the line separator. Perform a fuzzy search to determine the starting row of the field. and end line .
[0070] Step B: Anchor point location and coordinate fusion. If direct splicing and matching are difficult, an "anchor point" strategy is adopted. The longest substring with the highest distinguishability in Q is selected as anchor point A. First, A is located in the OCR, then the search extends to the left and right to find the remaining characters. Finally, the overall physical coordinates of the field are... It is calculated from the union of the bounding boxes of all covering characters: Explanation of formula variables: : Represents the set of all matched characters, which may span multiple lines of OCR text.
[0071] : The coordinates of the top-left corner of the c-th character in the set.
[0072] : The coordinates of the bottom right corner of the c-th character in the set.
[0073] The final calculated rectangle is the smallest bounding rectangle that can enclose the text across multiple lines (if it spans multiple lines, multiple rectangle lists can also be output).
[0074] [Application Scenario Example - Invoice]: The buyer's name is "Beijing XX Technology Development Co., Ltd." The OCR result is displayed in two lines: Line 10: "Name: Beijing XX Technology" Line 11: "Development Limited Company" LLM outputs the full name. The system first selects "Technology Development" as the anchor point, finding a match at the end of line 10 and the beginning of line 11. Then, the system locks the latter half of line 10 and the entirety of line 11, calculating using the formula described above. By merging the two lines of coordinates, the entire area containing the company name can be accurately selected.
[0075] Coordinate mapping and geometric calculation: Determine the logical position of a field in the OCR text (row index i, character range) After that, the system converts it into physical coordinates of the image.
[0076] Let the character-level coordinate sequence of the i-th line provided by the OCR engine be... ,in The bounding box is a single character. Then the overall coordinates of field Q are... Calculate the union of all character bounding boxes within the interval.
[0077] Through the above steps, the present invention achieves accurate backtracking from semantic fields to image pixel coordinates.
[0078] Output The system organizes the text value of each field and its corresponding coordinates on the image into the final output. For example, the output format could be a list of JSON objects, each containing a field_name, field_value, and bounding box (bbox coordinates). In this way, the entire system produces structured extraction results, along with the precise location of each field in the original image, which can be used for subsequent data validation, statistical analysis, or visualization annotation. Figure 3 As shown, this case extracts the project name, specifications, unit, quantity, unit price, amount, tax rate, and tax amount from the image. The upper half of the image is the input image, and the lower half is the extraction result of the large model. The gray-covered area in the image represents the tracing effect achieved using this method. The tracing result for the project name field demonstrates the effectiveness of this method for tracing cross-line scenarios, while the unit and tax rate fields demonstrate its effectiveness when multiple duplicate characters appear in the original text.
[0079] In summary, this invention discloses an information extraction and tracing method, system, and readable storage medium based on the collaboration of an OCR model and a large language model (LLM). It combines a small OCR model and a large language model (LLM) to extract textual information from images and trace back to the original image coordinates. This method enables structured extraction of textual information from images and precisely traces the extracted results to their location within the original image, achieving traceability and verifiability of the information.
[0080] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.
[0081] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.
[0082] Through the above description of the embodiments, those skilled in the art can clearly understand that the above implementation methods can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of the present invention.
[0083] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other modifications under the guidance of the present invention without departing from the spirit and scope of the claims. All of these modifications are within the protection scope of the present invention.
Claims
1. A method for information extraction and tracing based on the collaboration of OCR model and large language model, characterized in that, Includes the following steps: S1: Input the original image, and output the text content and corresponding coordinate information through the OCR model; S2: Concatenate the text content into a string and input it into the large language model to construct prompts for field extraction; S3: Match the fields extracted by the large language model with the text content output by the OCR model. If there is only one match, output it directly. If there are multiple matches, perform disambiguation based on the heuristic scoring mechanism of context semantics and spatial layout to determine the optimal matching result. If there are no matches, use the sliding window weighted edit distance algorithm for fault-tolerant matching. S4: Output the final matching result, including the fields and their corresponding coordinate information.
2. The information extraction and tracing method based on the collaboration of OCR model and large language model according to claim 1, characterized in that, In step S2, when concatenating the text content into a string, the text is concatenated line by line according to the reading order, with newline characters separating the lines.
3. The information extraction and tracing method based on the collaboration of OCR model and large language model according to claim 1, characterized in that, In S2, the extracted fields are output in JSON structure.
4. The information extraction and tracing method based on the collaboration of OCR model and large language model according to claim 1, characterized in that, S3 specifically includes: S31: The fields extracted by the large language model and the text content output by the OCR model are standardized and preprocessed; S32: For the standardized and preprocessed field, perform an exact substring search in the text content output by the OCR model. If there is only one match, proceed to S35; if there are multiple matches, proceed to S33; if there are no matches, proceed to S34. S33: Based on the heuristic scoring mechanism of contextual semantics and spatial layout, multiple matching items are scored separately for disambiguation, and the highest score is selected as the optimal matching result. Then, S35 is executed. S34: Use the sliding window weighted edit distance algorithm for fault-tolerant matching. Select the text content output by the OCR model with a similarity higher than the threshold as the optimal matching result, and then execute S35. S35: Determine the logical position of the field in the text content output by the OCR model and convert it into physical coordinates of the image; The order of steps 33 and 34 can be adjusted.
5. The information extraction and tracing method based on the collaboration of OCR model and large language model according to claim 4, characterized in that, In step S31, the standardization and preprocessing specifically include unifying font case and removing redundant whitespace symbols.
6. The information extraction and tracing method based on the collaboration of OCR model and large language model according to claim 4, characterized in that, In step S32, the field extracted by the large language model is Q, and the text content output by the OCR model is... The matching objective function is defined as finding the function that satisfies... The row index i and the character start index j, This represents the length of field Q.
7. The information extraction and tracing method based on the collaboration of OCR model and large language model according to claim 4, characterized in that, Specifically, S33 includes: Suppose that K candidate matches are found in the text output by the OCR model, denoted as set K. ,in This represents the row index of the k-th candidate match. This represents the starting character index of the k-th candidate match; The score for each candidate is calculated using the following formula. : in: : The total confidence score of the k-th candidate matching position; Weight hyperparameters are used to adjust the relative importance of context, layout, and text matching. Context similarity function, in candidate rows neighborhood Search for the "key" corresponding to this field. If it exists, the score for that item is close to 1; otherwise, it is close to 0. : Layout prior probability function, which defines the probability distribution of the position of a field based on the document type; if the position conforms to the prior distribution, a higher score is assigned; Edit the distance penalty item; The text content of the k-th candidate match; This is the length of field Q.
8. The information extraction and tracing method based on the collaboration of OCR model and large language model according to claim 4, characterized in that, In step S34, the field extracted by the large language model is Q, and each line of OCR text output by the OCR model... Construct a length of The sliding window, in which Initialized to the length of field Q ; During the sliding process, the similarity Sim between the substring sub and field Q within the window is calculated: The adaptive threshold determination logic is as follows: in: : Levinstein distance function, calculates the minimum number of single-character edit operations required to convert the substring sub into the field Q; : Normalization factor, ensuring similarity is within the interval [0, 1]; where, is the length of the substring sub; High confidence threshold: If the similarity is higher than this value, it is directly considered a successful match; The low confidence threshold, if it is in the middle range, needs to be further confirmed by combining the context score from the previous step to prevent false matching.
9. The information extraction and tracing method based on the collaboration of OCR model and large language model according to claim 4, characterized in that, In step S34, if there are no matches with similarity higher than the threshold, the anchor point extension algorithm is used to perform cross-line splicing and anchor point positioning, thereby selecting matches with similarity higher than the threshold as the optimal matching result, and then step S35 is executed.
10. The information extraction and tracing method based on the collaboration of OCR model and large language model according to claim 9, characterized in that, The anchor point extension algorithm specifically includes: Cross-line text aggregation: Detecting the length of field Q extracted from a large language model ,like If the number of characters exceeds the average number of characters in a single line of OCR, then a multi-line concatenated string is constructed. : in, For a multi-line text block concatenated starting from line k, the string is composed of... It is pieced together; This represents a string concatenation operation. As a line separator; in Perform a fuzzy search to determine the starting row of field Q. and end line ; Anchor point location and coordinate fusion: If direct splicing and matching are difficult, an "anchor point" strategy is adopted. The longest substring with the highest distinguishability in Q is selected as anchor point A. First, A is located in the text content output by the OCR model, and then the search is expanded to the left and right to find the remaining characters. Finally, the overall physical coordinates of the field are fused. It is calculated from the union of the bounding boxes of all covering characters: in: : Represents the set of all matched characters, which spans multiple lines of OCR text; The coordinates of the top-left corner of the c-th character in the set; The coordinates of the bottom right corner of the c-th character in the set; The final calculated rectangle is the smallest bounding rectangle that can enclose the multi-line text.
11. The information extraction and tracing method based on the collaboration of OCR model and large language model according to claim 4, characterized in that, Specifically, S35 is as follows: Let the character-level coordinate sequence of the i-th row provided by the OCR engine be... ,in A bounding box for a single character. Then the overall coordinates of field Q. Calculate the union of all character bounding boxes within the interval.
12. The information extraction and tracing method based on the collaboration of OCR model and large language model according to claim 1, characterized in that, In S4, the output format is a list of JSON objects, each object containing a field name, field value, and bounding box coordinates.
13. An information extraction and tracing system based on the collaboration of an OCR model and a large language model, the system comprising: A processor and a memory for storing executable instructions; characterized in that the processor is configured to execute the executable instructions to perform the information extraction and tracing method based on a native multimodal large model as described in any one of claims 1 to 12.
14. A computer-readable storage medium, characterized in that, It stores a computer program, which, when executed by a processor, implements the information extraction and tracing method based on the collaboration of an OCR model and a large language model as described in any one of claims 1 to 12.