A multi-engine OCR result fusion and reading order optimization method, system, device and storage medium

Abstract

The application relates to the technical field of document intelligent processing and computer vision information processing, and discloses a multi-engine OCR result fusion and reading order optimization method, system, device and storage medium. The method receives original document images and heterogeneous recognition results of at least two OCR engines, uniformly analyzes and generates a standardized structured text element set, constructs a cross-engine weighted bipartite graph, combines geometric positions and text contents to complete candidate matching and edge screening, obtains matching groups and isolated text elements through connected domain analysis and community division, reconstructs a global unified reading order chain according to layout position relationships, constructs a constraint prompt word and candidate text set based on the reading order chain, inputs a large model to execute controlled optimization decision, and finally encapsulates and outputs structured data and readable text dual-type results. The application can improve the fusion accuracy of multi-engine OCR results of complex layout documents, the recovery accuracy of reading orders and the traceability of output results.

Description

Technical Field

[0001] This invention relates to the fields of intelligent document processing and computer vision information processing technology, and in particular to a method, system, device and storage medium for multi-engine OCR result fusion and reading order optimization. Background Technology

[0002] In engineering applications such as document digitization, intelligent parsing of certificates and invoices, and structured extraction of page content, optical character recognition (OCR) technology has become a core supporting means for converting paper documents into electronic text. OCR engines from different vendors and with different architectures exhibit varying performance in character recognition accuracy, page segmentation granularity, table reconstruction capabilities, and resistance to complex noise. A single OCR engine cannot simultaneously adapt to the recognition needs of complex formats such as multi-column documents, nested tables, and certificates and invoices. In actual business, multi-engine parallel recognition is often used to improve content coverage; however, the outputs of different engines exhibit inconsistent text segmentation granularity, bounding box position offsets, and inconsistent original sorting logic. The recognition results of heterogeneous formats are difficult to directly reuse and integrate, becoming a common technical bottleneck in the intelligent structuring of documents.

[0003] Current multi-engine OCR result fusion methods mostly employ character-level voting comparison and independent text similarity matching, lacking the ability to jointly model the geometric position of the layout and the semantic relationship of the text. Conventional techniques cannot effectively identify complex text matching relationships such as one-to-one, one-to-many, many-to-one, and many-to-many, and struggle to handle the corresponding ambiguities caused by the splitting and merging of text blocks from different engines. Traditional reading order restoration often relies on fixed template constraints or a single heuristic top-down and left-to-right sorting logic, which cannot adapt to non-standard documents and invoice / table scenarios with varying layouts. At the same time, existing solutions often directly rely on large models to freely generate full-text content, which is prone to semantic arbitrariness, missing fields, and high inference latency. Furthermore, they lack a unified structured intermediate representation, requiring downstream information extraction modules to be adapted separately for different OCR engines, resulting in high system maintenance and expansion costs. Summary of the Invention

[0004] This invention discloses a method for multi-engine OCR result fusion and reading order optimization, comprising the following steps: S1. Receive the original document image and the heterogeneous recognition results of at least two OCR engines, uniformly parse and generate a standardized set of structured text elements; S2. Construct a cross-engine weighted bipartite graph based on the structured text element set, and combine geometric position features and text content features to complete element candidate matching and edge filtering; S3. Perform connected component analysis and community partitioning on the weighted bipartite graph to obtain matching groups and isolated text elements; S4. Based on the page position relationship between the matching group and the isolated text element, reconstruct the globally unified reading order chain; S5. Based on the reading order chain, construct a set of constrained prompt words and candidate texts to form a large model's constrained input data; S6. Input the restricted input data into the large model to perform controlled optimization decision-making and output standardized text content; S7. Based on the standardized text content and full-link metadata, encapsulate and generate dual-type output results of structured data and readable text.

[0005] Preferably, step S1 includes defining a single text element as having at least the following fields: text content, bounding box coordinates, page number, element type, source engine identifier, original sequence index, identification confidence level, and page level identifier, and classifying the element type into at least one of the following: body text, title, table cell, header / footer, key-value pair field, and signature area.

[0006] Preferably, step S2 includes: calculating normalized text similarity based on the edit distance between the text elements to be matched and the text length, wherein the normalized text similarity includes calculation based on edit distance, n-gram similarity, character-level vector similarity and field pattern similarity; The comprehensive matching weight is calculated based on the bounding box overlap and the normalized text similarity. For table areas or areas without valid text content, reduce or set the contribution of text similarity to the overall matching weight.

[0007] Preferably, step S2 further includes setting a lower bound threshold for bounding box overlap and a lower bound threshold for comprehensive matching weight, and retaining only candidate edges that simultaneously satisfy the lower bound threshold for bounding box overlap and the lower bound threshold for comprehensive matching weight.

[0008] Preferably, step S3 includes: performing connected component retrieval on the weighted bipartite graph; performing community partitioning on connected subgraphs with a number of nodes exceeding a preset scale threshold using a preset community partitioning algorithm, wherein the community partitioning algorithm includes any one of Louvain algorithm, Leiden algorithm, spectral clustering, and threshold-based hierarchical clustering; determining the matching type (one-to-one, one-to-many, many-to-one, and many-to-many) based on the number relationship between nodes corresponding to different OCR engines within each community, and calculating the matching quality score based on the comprehensive matching weight of candidate edges within the community; and marking nodes that have not formed a valid pair as isolated text elements.

[0009] Preferably, step S4 includes: initially sorting the elements on both sides according to the original index and page coordinates, and grouping and merging the matches; performing a two-pointer traversal based on the merged matching group results, and remapping the traversal output to the final reading order index; sorting the ordinary page text elements according to the rule of vertical coordinate priority and horizontal coordinate priority; first completing local matching and logical ordering of the cells in the table area, and then merging the local table order into the global reading order chain; and connecting isolated text elements to the global reading order chain according to their relative positions in the original page.

[0010] Preferably, step S5 includes: limiting the large model to select text content only from the given candidate text set in the prompt words, performing only a small amount of error correction and formatting, outputting according to the reading order chain, and prohibiting autonomous generation outside the candidate text set.

[0011] Preferably, step S6 includes: performing a comprehensive comparison and screening of the multiple candidate texts for each reading unit based on semantic integrity, character accuracy, and format standardization to determine the optimal benchmark text; performing redundant character removal, space normalization, punctuation correction, and content normalization on the optimal benchmark text; and not introducing new text content outside the scope of the candidate text set during the entire optimization process.

[0012] Preferably, step S7 includes: generating structured JSON data including page number, reading order index, standardized text, bounding box, source engine identifier, matching type and confidence level; and synchronously generating sequential Markdown readable text according to the reading order chain; while retaining full-link traceability metadata for audit review.

[0013] Preferably, the feature weight coefficients and candidate edge filtering thresholds in the comprehensive matching weights are adaptively configured according to the document layout type, element type, or region type.

[0014] This invention also discloses a multi-engine OCR result fusion and reading order optimization system, including an input module, a parsing module, a graph matching module, a community division module, a reading order module, a prompt word construction module, a large model optimization module, and an output module. The modules work together to execute the above-mentioned multi-engine OCR result fusion and reading order optimization method.

[0015] The present invention also discloses an electronic device, including a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, it implements the above-mentioned method for multi-engine OCR result fusion and reading order optimization.

[0016] The present invention also discloses a computer-readable storage medium storing a computer program thereon, wherein when the computer program is executed by a processor, it implements the above-described method for multi-engine OCR result fusion and reading order optimization.

[0017] Compared with the prior art, the present invention has at least the following beneficial effects: This invention constructs a weighted bipartite graph matching mechanism that fuses geometric and textual features, and combines community partitioning to complete the recognition of complex matching relationships. It can break away from the limitations of a single OCR engine, achieve accurate alignment and fusion of text elements output by different recognition engines, and improve the integrity and semantic consistency of text recognition content in complex document layouts.

[0018] This invention reconstructs a globally unified reading order chain based on cross-engine semantic alignment. It adopts differentiated sorting logic for ordinary layouts and table areas to avoid the order disorder defects caused by fixed sorting rules, ensure the accuracy of the natural reading logic of multi-column invoice nested table documents, and reduce the parsing error of downstream field extraction and document understanding tasks.

[0019] This invention limits the large model to a working mode of candidate selection and controlled error correction optimization. It relies on constraint prompts to standardize the output boundary of the large model. While leveraging the semantic discrimination ability of the large model, it suppresses the illusion problem caused by the generation of unfounded content. At the same time, it outputs two types of results: traceable structured data and readable text. This reduces the coupling dependence of downstream systems on different OCR output formats and lowers the system adaptation and maintenance expansion costs. Attached Figure Description

[0020] Figure 1 This is a flowchart illustrating the overall process architecture of the multi-engine OCR result fusion and reading order optimization method of the present invention. Detailed Implementation

[0021] To enable those skilled in the art to better understand the technical solutions of this invention, the technical solutions in the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings and specific examples. It should be noted that, unless otherwise specified, the embodiments and technical features in these embodiments can be combined with each other.

[0022] Example 1 See Figure 1As shown, this embodiment provides a method, system, device, and storage medium for multi-engine OCR result fusion and reading order optimization. The overall process of the multi-engine OCR result fusion and reading order optimization method includes: S1, receiving the original document image and heterogeneous recognition results from at least two OCR engines, and uniformly parsing to generate a standardized set of structured text elements; S2, constructing a cross-engine weighted bipartite graph based on the structured text element set, and combining geometric position and text content to complete element candidate matching and edge filtering; S3, performing connected component analysis and community partitioning on the weighted bipartite graph to obtain multiple types of matching groups and isolated text elements; S4, reconstructing a globally unified reading order chain based on the page position relationship between the matching groups and isolated text elements; S5, constructing constrained prompt words and candidate text sets based on the reading order chain, and constructing large-scale model constrained input data; S6, inputting the constrained input data into the large-scale model to perform controlled optimization decisions and output standardized text content; S7, encapsulating and generating dual-type output results of structured data and readable text based on the standardized text content and full-link metadata.

[0023] Step S1 further includes S1-1 Standardized field definition for text elements. The definition of a single text element includes text content, bounding box coordinates, page number, element type, source engine identifier, original order index, identification confidence, and page level identifier fields. The element type classification includes body text, title, table cell, header and footer, key-value pair fields, and signature area.

[0024] Step S2 further includes S2-1, calculating the dual-dimensional similarity and comprehensive matching weight, and constructing a normalized text similarity formula: Construct a matching weight formula that integrates geometric and textual features: in, For text editing distance, , For the two sets of text elements to be matched, , These represent the lengths of the corresponding text characters. Used for linear normalization of edit distance; This represents the overlap of the bounding boxes of the candidate elements; , Configurable weight parameters; This represents the normalized similarity of candidate texts.

[0025] In this embodiment, linear normalized edit distance is used to calculate text similarity. The ratio of text edit distance to the maximum length of the two texts is used as the normalization benchmark to eliminate the measurement bias between long and short texts.

[0026] Step S2 further includes S2-2 candidate edge threshold filtering, setting a fixed IoU lower bound threshold and a comprehensive matching weight lower bound threshold, and retaining bipartite graph candidate edges that simultaneously meet both threshold conditions.

[0027] Step S3 further includes S3-1 graph community partitioning and matching type determination. The Louvain algorithm is used to partition the connected components of the weighted bipartite graph into communities. Based on the number of engine nodes on both sides within the community, one-to-one, one-to-many, many-to-one, and many-to-many matching types are obtained. The matching quality score is calculated simultaneously and isolated text elements are marked.

[0028] Step S4 further includes S4-1 constructing a reading order by region. The normal layout is sorted according to the rule of vertical coordinate priority and horizontal coordinate second. For the table area, local matching and logical ordering of cells are completed first, and then the local order is incorporated into the global reading order chain. Unmatched isolated elements are connected to the reading order chain according to their relative position on the original layout.

[0029] Step S5 also includes S5-2 setting large model behavior constraints, which restricts the large model to only select the best candidate text in the prompt words, perform only a small amount of error correction and formatting, output according to the established reading order, and prohibits the independent generation and creation outside the candidate text.

[0030] Step S6 calls the prompt words and candidate text set generated in step S5 as input, and completes text selection, redundancy removal and content normalization processing under constraints.

[0031] Step S7 further includes S7-1 constructing a dual-format traceable output, generating structured JSON data including page number, reading order index, standardized text, bounding box, source engine, matching type, and confidence level, synchronously generating sequential Markdown readable text, and retaining full-link traceability metadata for audit review.

[0032] In the overall implementation process of this embodiment, step S1 is first executed to complete the raw data access and structured normalization processing. The external input data stream includes the original document image carrier and the recognition results of at least two OCR engines in different formats. The recognition result formats include JSON, Markdown, hOCR, and ALTO XML. The system's built-in unified parsing engine performs syntax parsing and field mapping on the heterogeneous format data. The parsing process strictly follows the text element standardized field definition rules set in S1-1, abstracting each original text block into a structured text element unit with multi-dimensional attributes. All parsed text elements are aggregated to form a global structured text element set, which serves as the basic input data source for the subsequent graph structure modeling in step S2. All elements retain the original page position, engine identifier, and page level attributes, preserving the original feature basis for subsequent geometric matching and sequential reconstruction.

[0033] The parsed set of structured text elements is directly fed into step S2 for cross-engine weighted bipartite graph construction. All text elements output by the first OCR engine are used as the first side node set of the bipartite graph, and all text elements output by the second OCR engine are used as the second side node set of the bipartite graph. All pairwise combinations of the first and second side nodes are traversed to form candidate pairing relationships. For each candidate pairing, the dual-dimensional similarity and comprehensive matching weight calculation logic of S2-1 is called to complete the quantitative solution of text similarity and comprehensive matching weight in sequence.

[0034] Text similarity calculation is performed using a modified exponential decay formula. Text edit distance is used as the basic difference metric. Normalization constraints are introduced using text character length and a fixed lower bound constant to eliminate measurement bias between long and short texts. The matching weight calculation integrates two feature components: bounding box overlap and exponential text similarity. The weight ratio of each feature dimension is adjusted through two sets of independent weight coefficients. The weight coefficients can be adaptively configured according to the document layout type to ensure a reasonable distribution of feature contribution under different document layouts.

[0035] After all weight calculations are completed, the S2-2 candidate edge threshold screening mechanism is executed. Invalid pairs with low geometric overlap are filtered out using a preset IoU lower bound threshold, and weakly associated pairs with substandard feature fusion matching degree are eliminated using a comprehensive matching weight lower bound threshold. Only candidate pairs that meet both thresholds are retained as valid candidate edges in the bipartite graph, which simplifies the graph structure and eliminates meaningless matching associations, reducing the computational load of subsequent graph partitioning algorithms.

[0036] The weighted bipartite graph with valid candidate edges output in step S2 is directly fed into step S3. The system first performs a global connectivity search on the weighted bipartite graph to identify interconnected subgraphs of nodes in the graph. For connected subgraphs with more than a preset scale threshold, the Louvain algorithm is used to perform community partitioning operations. The Louvain algorithm relies on the modularity maximization criterion to complete local community aggregation and global community structure optimization, decomposing complex connected subgraphs into several independent community units with consistent semantic association.

[0037] After completing the community division, the number of first-side engine nodes and second-side engine nodes within each community unit is counted. The matching type is determined based on the correspondence between the number of nodes on both sides. The matching type is divided into four types: one-to-one, one-to-many, many-to-one, and many-to-many. At the same time, the matching quality score is calculated based on the average of the comprehensive matching weights of all candidate edges within the community. This score serves as the confidence basis for subsequent reading order ranking and large model candidate selection. For isolated nodes that cannot form an effective pair with any node on the other side, they are uniformly marked as text elements unique to a single engine and stored separately. This achieves the complete decomposition and classification of complex correspondences between text elements of multiple engines.

[0038] All matching groups, matching quality scores, and isolated text elements output in step S3 are incorporated into the reading order chain construction process in step S4, and the regional reading order construction logic in S4-1 is executed to achieve global order reconstruction.

[0039] The system initially sorts the elements on both sides according to the original index and page coordinates, and performs grouping and merging for 1:N, N:1 and N:N matching. Then, it performs two-pointer traversal based on the merging results and remaps the traversal output to the final reading order index. For text elements in a regular layout, a coarse sort is performed based on the vertical coordinates of the bounding box, and then a fine sort is performed based on the horizontal coordinates within the same vertical coordinate range. For various matching groups, such as one-to-one, one-to-many, many-to-one, and many-to-many, elements within the group are merged and integrated, and text elements from multiple engines under the same semantic unit are aggregated into a sequential processing unit. For the table-specific areas identified within the document, a local priority processing strategy is implemented. First, the matching and grouping of cells within the table area are completed and arranged in the logical order of the cells, which follows the general business reading logic. Then, the complete sequential units of the table are embedded into the global layout position sequence to avoid the interference of nested table layouts on the global reading order. For isolated text elements marked in step S3, their relative arrangement in the initial layout sequence is not changed. They are accessed into the global sequence as independent sequential units. Finally, all matching group units, table local sequential units, and isolated element units are integrated to generate a globally unified reading order chain with natural document reading logic. Each sequential unit is bound to the corresponding candidate text set and matching type identifier.

[0040] The reading order chain generated in step S4 serves as the core input carrier for step S5. The system traverses each independent reading unit in the reading order chain, extracts the multi-engine candidate text content included in the unit, and performs structured prompt word assembly by combining the document context layout information and output format specification requirements. The prompt word structure includes a candidate text list, context semantic description, and constraints such as output format and behavior rules.

[0041] At the same time, it strictly follows the S5-2 large model behavior constraint setting requirements, clearly defines the working boundary of the large model in the prompt text, limits the model to select the optimal content only within the provided candidate text set during the operation process, allows minor error correction and formatting of candidate texts with character defects and format disorder, and mandates that the model output content strictly follow the constructed reading order chain arrangement logic, completely prohibits the model from making unfounded autonomous expansions and content fabrications outside the given candidate text, avoids the large model illusion problem from the input level through explicit rule constraints, and the assembled prompt words and corresponding candidate text sets together constitute the restricted input data of the large model.

[0042] The restricted input data constructed in step S5 is directly sent to the large model optimization module in step S6. The large model uses the restricted input data as the sole reasoning basis, without introducing additional external knowledge bases or irrelevant text information. It performs a comprehensive comparison of multiple candidate texts for each reading unit in terms of semantic completeness, character accuracy, and format standardization to determine the optimal benchmark text. Based on this, redundant characters, abnormal spaces, and incorrect punctuation in the base text are automatically cleaned up to complete the content normalization process. Throughout the process, the system strictly adheres to the constraints set by the prompt words, does not generate content that exceeds the candidate range, outputs standardized text content corresponding to each reading unit, and retains the matching type and source engine traceability information corresponding to each piece of content.

[0043] The standardized text content output in step S6 and the metadata information retained throughout the process are sent to step S7, executing the S7-1 dual-format traceable output construction mechanism. On the one hand, JSON format data is generated according to the preset structured field specifications. The JSON data fully carries all metadata information, including page number, reading order index, standardized text content, bounding box coordinate parameters, source engine identifier, matching type classification, and matching confidence score, meeting the structured calling requirements of downstream machine automated parsing, field extraction, and data storage. On the other hand, the standardized text is spliced ​​and formatted according to the order of the reading order chain to generate human-readable Markdown text, which is suitable for business scenarios such as manual document review, content archiving, and knowledge organization. The system synchronously retains the full-link processing logs and traceability metadata, supporting the retrospective analysis needs of subsequent manual error location, OCR engine recognition quality assessment, and incremental optimization of algorithm parameters, realizing bidirectional adaptation between machine structured applications and manual review and archiving.

[0044] In this embodiment, from the generation of the structured text element set to the construction of the bipartite graph and the division of the community, and then to the reconstruction of the reading order chain and the controlled optimization of the large model, the data is transferred step by step and the features are reused step by step, forming a complete closed-loop logic.

[0045] In this embodiment, the algorithm logic and field definitions throughout the entire process remain consistent, ensuring the accuracy and traceability of multi-engine OCR result fusion and reading order optimization.

[0046] Example 2 This embodiment, combined with the complex layout application scenario of invoices and receipts, provides a method for multi-engine OCR result fusion and reading order optimization. The overall process of the method includes: S1, receiving the original invoice document image and the heterogeneous recognition results of at least two OCR engines, and uniformly parsing to generate a standardized set of structured text elements; S2, constructing a cross-engine weighted bipartite graph based on the structured text element set, and completing element candidate matching and edge filtering by combining geometric position and text content; S3, performing connected component analysis and community partitioning on the weighted bipartite graph to obtain multiple types of matching groups and isolated text elements; S4, reconstructing the globally unified reading order chain of the invoice document based on the layout position relationship of the matching groups and isolated text elements; S5, constructing a constrained prompt word and candidate text set based on the reading order chain, and constructing a large model constrained input data; S6, inputting the constrained input data into the large model to perform controlled optimization decisions, and outputting standardized text content of invoice fields; S7, based on the standardized text content of invoice fields and full-link metadata, encapsulating and generating dual-type output results of invoice structured data and readable text.

[0047] Step S1 further includes S1-1 Standardized field definition of text elements, defining a single invoice scenario text element including text content, bounding box coordinates, page number, element type, source engine identifier, original sequence index, identification confidence, and page level identifier fields. Element type classification includes body text, title, table cell, header and footer, key-value pair fields, and signature area. In the invoice scenario, the amount field, tax number field, date field, and merchant information field are marked with independent element types.

[0048] Step S2 further includes S2-1, calculating the dual-dimensional similarity and comprehensive matching weight, and constructing a normalized text similarity formula: Construct a matching weight formula that integrates geometric and textual features: in, For text editing distance, , For the two sets of text elements to be matched, , These represent the lengths of the corresponding text characters. Used to linearly normalize the edit distance; This represents the overlap of the bounding boxes of the candidate elements; , Configurable weight parameters; This represents the normalized similarity of candidate texts.

[0049] Step S2 further includes S2-2 candidate edge threshold screening, setting an IoU lower bound threshold and a comprehensive matching weight lower bound threshold to adapt to the bill layout, retaining bipartite graph candidate edges that simultaneously meet the two threshold conditions, weakening the contribution of text similarity weight to the bill amount area and table detail area, and strengthening the dominant role of matching geometric position and layout offset score.

[0050] Step S3 further includes S3-1 graph community partitioning and matching type determination. The Louvain algorithm is used to partition the connected domain of the weighted bipartite graph into communities. Based on the number of engine nodes on both sides within the community, one-to-one, one-to-many, many-to-one, and many-to-many matching types are obtained. The quality score of the bill scene matching is calculated simultaneously and isolated text elements are marked. The bill details line has the feature of single-line splitting into multiple lines for recognition. One-to-many and many-to-many complex matching group structures are identified first.

[0051] Step S4 further includes S4-1 constructing a regional reading order, sorting the ordinary text page of the invoice according to the rule of vertical coordinate priority and horizontal coordinate second, first completing local matching and cell logical ordering of the invoice detail table area, arranging the internal cell order of the table according to the row priority logic, and then merging the local order of the table into the global reading order chain, and unmatched isolated elements are connected to the reading order chain according to the relative position of the original page of the invoice.

[0052] Step S5 also includes setting large model behavior constraints in S5-2. The prompt word restricts the large model to select the best candidate text for the invoice only, and performs only a small amount of text correction, amount format regularization, and date format standardization processing. It follows the established invoice reading order chain for output and prohibits the model from generating its own content outside the candidate text.

[0053] Step S6 calls the invoice scenario prompts and candidate text set generated in step S5 as input, and completes the invoice text selection, redundant invoice character removal and field content normalization processing under constraints.

[0054] Step S7 further includes S7-1 constructing a dual-format traceable output, generating structured JSON data including page number, reading order index, standardized invoice text, bounding box, source engine, matching type, and confidence level, synchronously generating sequential Markdown readable invoice text, and retaining full-link traceability metadata for manual review and field validation of invoices.

[0055] In the complete implementation of this invoice scenario, step S1 is first executed to complete the access and structured normalization of the original invoice data. The external input data stream includes the original document image generated by scanning the paper document of the invoice, as well as the heterogeneous recognition results of at least two different OCR engines adapted to invoice recognition. The recognition result formats include JSON, Markdown, hOCR, and ALTO XML.

[0056] The system has a built-in unified parsing engine that performs syntax parsing and field mapping for the special format features of invoices. The parsing process strictly follows the standardized field definition rules for text elements in the invoice scenario set in S1-1, and abstracts the merchant name, transaction date, tax number information, total amount, detailed list, signature and remarks, etc. in the invoice into structured text element units with multi-dimensional attributes.

[0057] All parsed document text elements are aggregated to form a global structured text element set. This set directly serves as the sole basic input data source for the subsequent step S2 to construct the document scene cross-engine graph structure. Each document text element fully retains its original page coordinate position, its own OCR engine identifier, and its document-specific layout hierarchy attributes, thus preserving original features to support subsequent document geometric matching, field merging, and reading order reconstruction.

[0058] The parsed set of structured text elements of the invoice is sent to step S2 to construct a cross-engine weighted bipartite graph. All text elements output by the first invoice-specific OCR engine are defined as the first side node set of the bipartite graph, and all text elements output by the second invoice-specific OCR engine are defined as the second side node set of the bipartite graph.

[0059] The system iterates through all pairs of nodes on both sides to form candidate pairings of bill texts. For each pair of bill candidate pairings, it calls the S2-1 dual-dimensional similarity and comprehensive matching weight calculation logic to complete the bill text similarity quantification and multi-feature fusion matching weight solution in sequence.

[0060] The similarity calculation of invoice text follows the linear normalized edit distance formula, using the edit distance of invoice characters as the basic difference measure, and introducing the character length of invoice text as a normalization constraint to effectively avoid the measurement imbalance problem between short fields and long detailed text of invoices.

[0061] The matching weight calculation continuously integrates two types of feature components: bounding box overlap and linearly normalized text similarity. It adapts the feature contribution ratio of different regions of the invoice through two sets of independent weight coefficients. The text similarity coefficient value is increased for pure text fields on the invoice, while the text similarity coefficient value is reduced for table details, amount values, and border nested regions on the invoice. The matching logic is dominated by geometric overlap.

[0062] After all weight calculations are completed, the S2-2 candidate edge threshold screening mechanism is executed. A dedicated IoU lower bound threshold adapted to the characteristics of the invoice format is used to filter invalid invoice text pairs with low overlap. A comprehensive matching weight lower bound threshold specific to the invoice scenario is used to eliminate weakly associated pairs. Only candidate pairs that meet both thresholds are retained as valid candidate edges in the bipartite graph. This greatly simplifies the redundant associations in the graph structure under the complex invoice format and reduces the computational overhead and mismatch probability of the subsequent community partitioning algorithm.

[0063] The weighted bipartite graph of the bill scenario with valid candidate edges output in step S2 is directly passed to step S3. The system first performs a full-domain connected component traversal search on the weighted bipartite graph to identify all connected subgraphs that are interconnected between the bill text nodes.

[0064] For connected subgraphs with a large number of nodes formed by the bill details area and multi-column layout area, the Louvain algorithm is used to perform community partitioning operations. The algorithm relies on the principle of maximizing modularity to complete the aggregation of local communities and the optimization of global community structure, decomposing the originally mixed and related connected subgraphs of bills into several semantically independent community units with clear field affiliations.

[0065] After completing the community division of the invoice scenario, the number of nodes of the OCR engines on both sides within each community unit is counted. The matching type is strictly divided according to the corresponding relationship of the number of nodes. In the invoice scenario, the detailed line splitting and recognition, the amount line display, and the remarks field splitting and description form one-to-many, many-to-one, and many-to-many matching group structures, while the regular fixed fields form one-to-one matching group structures.

[0066] The system calculates a unique matching quality score for each ticket based on the average of the comprehensive matching weights of all valid candidate edges within the community. This score is used as the basis for determining the confidence level of subsequent ticket reading order and candidate selection of large model fields. For unique text elements of tickets that cannot form a valid pair with the other side engine node, they are uniformly marked as isolated nodes and stored independently, realizing the complete decomposition and regular classification of complex correspondences of text elements of multiple engines in tickets.

[0067] The various ticket matching groups, matching quality scores, and isolated ticket text elements output in step S3 are fully integrated into the global ticket reading order chain construction process in step S4, and the regional reading order construction logic in S4-1 is strictly executed to complete the reconstruction of the ticket layout order.

[0068] The system first performs an initial sorting of the ticket elements on both sides according to their original indexes and page coordinates; secondly, it groups and merges 1:N, N:1, and N:N matches; then, it performs a two-pointer traversal based on the grouping and merging results, and remaps the traversal output to the final reading order index; for ordinary page text elements, it performs a global coarse sorting based on the vertical coordinates of the bounding box, and then completes a fine arrangement according to the horizontal coordinates within the same vertical height range.

[0069] For various types of invoice matching groups, such as one-to-one, one-to-many, many-to-one, and many-to-many, the text elements of multiple engines within the group are merged and integrated, and the multi-version recognition content under the same invoice semantic field is aggregated into a single sequential processing unit.

[0070] The system employs a local priority processing strategy for the table detail areas automatically identified and extracted from the invoice document. First, it completes the cell matching group merging within the table, arranges the cell local reading order according to the general row priority logic of financial invoices, and then seamlessly embeds the complete table local sequence unit into the global page position sequence of the invoice, avoiding the disruption of the global reading logic caused by invoice table nesting and multi-column cross layout.

[0071] For isolated text elements of the invoice marked in step S3, their relative arrangement in the initial page position sequence remains unchanged. They are then accessed in the global sequence as independent sequential units. Finally, all invoice matching group units, table local sequential units, and isolated field units are integrated to generate a globally unified reading sequence chain of invoices that conforms to human review habits and financial archiving standards. Each sequential unit is synchronously bound to the corresponding invoice candidate text set and matching type identifier.

[0072] The document reading sequence chain generated in step S4 serves as the core input carrier for step S5. The system traverses each independent document reading unit within the reading sequence chain, extracts candidate text content output by multiple OCR engines within the unit, and performs standardized assembly of structured prompt words in conjunction with the document context layout structure, financial field output format specifications, and archived text layout requirements.

[0073] The prompt text includes several fixed components, such as a list of candidate document texts, semantic descriptions of field context, structured output format constraints, and large model behavior rules. It also strictly adheres to the S5-2 large model behavior constraints, clearly defining the execution boundaries of the large model in the prompt text.

[0074] The model's computation process only allows the selection of optimal field content within a given set of candidate document texts. It allows for minor error correction and format standardization for issues such as similar-looking character errors, incorrect punctuation of amounts, and inconsistent date formats that arise during document recognition. The model's output arrangement is forced to strictly follow the logical sequence of the established document reading order. It completely prohibits the model from adding content, fabricating fields, or altering semantics without basis, detached from the candidate document texts. Through explicit rule constraints, it suppresses the interference caused by the illusion of a large model on document field extraction from the input source. The assembled prompt words and the set of candidate document texts together constitute the restricted input data for the large model.

[0075] The restricted input data of the invoices constructed in step S5 is directly sent to the large model optimization module in step S6. The large model uses only the restricted input data as the sole reasoning basis and does not introduce external invoice dictionaries or irrelevant business text information.

[0076] The model sequentially compares and filters multiple versions of candidate text for each invoice reading unit based on semantic integrity, character recognition accuracy, and financial format standardization to determine the optimal baseline field text. Then, it automatically removes redundant spaces, invalid symbols, and incorrect punctuation in the baseline text and normalizes the format to unify the standard writing format of invoice date, amount, and tax number.

[0077] The model adheres to the constraints set by the prompt words throughout the process, does not generate autonomous content beyond the scope of the candidate text, outputs standardized invoice text content unit by unit, and retains the matching type, source engine traceability identifier and matching confidence information corresponding to each field content.

[0078] The standardized invoice text content output in step S6, along with the format metadata and matching metadata retained throughout the process, are uniformly sent to step S7 to execute the S7-1 dual-format traceable output construction mechanism.

[0079] On the one hand, in accordance with the preset structured field specifications for the financial system, standard JSON format invoice data is generated. The JSON structure fully carries all metadata, including page number, invoice reading order index, standardized field text, bounding box coordinate parameters, source engine identifier, matching type classification, and matching confidence score, meeting the machine-structured call requirements of the financial system for automatic invoice entry, field extraction, ledger archiving, and data retrieval. On the other hand, according to the predetermined order of the invoice reading order chain, the standardized invoice field text is spliced ​​into paragraphs and adaptively formatted to generate human-readable invoice documents in Markdown format, adapting to application scenarios such as manual invoice review, voucher archiving, and business review in finance.

[0080] The system synchronously retains traceability metadata such as the entire process log of invoices, matching process records, and large model optimization trajectory. It supports retrospective analysis of subsequent manual invoice error location, multi-OCR engine invoice recognition quality evaluation, and incremental optimization of algorithm weight parameters. At the same time, it provides auditable and complete data link support for the intelligent invoice archiving system.

[0081] This invoice scenario implementation fully reuses all steps, formula models, algorithm mechanisms, and innovative sub-steps of the main method. Only the element type division, weight coefficient adaptation, threshold parameter setting, local sorting rules, and large model constraint details are refined and adapted to fit the characteristics of invoice business. Throughout the process, all technical terms, formula symbols, and parameter interpretations are kept completely consistent with the previous text, and there are no polysemous symbols or conflicts in technical logic.

[0082] Each step maintains a tight data dependency and business interaction logic between the invoices. The set of structured text elements of the invoices generated by step S1 provides the basic data source for graph node modeling in step S2. The weighted bipartite graph of invoices and effective candidate edges output by step S2 provide topological support for community partitioning in step S3. The invoice matching groups and isolated fields classified in step S3 provide semantic grouping basis for sequential reconstruction in step S4. The invoice reading order chain constructed in step S4 provides unit partitioning and order constraint benchmark for prompt word assembly in step S5. The restricted input data of invoices generated in step S5 provides constraint boundaries and candidate materials for field optimization of the large model in step S6. The standardized invoice field text output by step S6 provides the core content source for dual-format result encapsulation in step S7.

[0083] The entire process of invoice features is reused at each level and data is transferred at each level. Relying on the established innovative technology framework, it achieves accurate fusion of multi-engine OCR results under complex invoice formats, automatic correction of reading order, and standardized output of financial fields, which is fully adapted to the actual engineering implementation needs of digital archiving, intelligent review, and structured extraction of invoices.

[0084] In one feasible embodiment, the number of OCR engines is not limited to two, and a one-main-multiple-auxiliary or multi-engine anchoring and fusion strategy can be adopted. In one feasible embodiment, the bounding box can be represented by a rectangular box, a four-point box, or a normalized coordinate box; In one feasible embodiment, the community partitioning algorithm is not limited to Louvain, but may also employ Leiden, spectral clustering, or threshold-based hierarchical clustering; In one feasible embodiment, text similarity is not limited to edit distance, but may also employ n-gram similarity, character-level vector similarity, or field pattern similarity; In one feasible embodiment, the output is not limited to JSON and Markdown, but can also be extended to XML, database records, knowledge graph nodes or business system interface formats; In one feasible embodiment, the LLM large model can be deployed as a cloud model or a private model, or it can be replaced with other language models or multimodal models with candidate discrimination capabilities.

[0085] It should be understood that the above embodiments are only for illustrating the technical solutions of the present invention, and not for limiting them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for fusing multi-engine OCR results and optimizing reading order, characterized in that, Includes the following steps: S1. Receive the original document image and the heterogeneous recognition results of at least two OCR engines, uniformly parse and generate a standardized set of structured text elements; S2. Construct a cross-engine weighted bipartite graph based on the structured text element set, and combine geometric position features and text content features to complete element candidate matching and edge filtering; S3. Perform connected component analysis and community partitioning on the weighted bipartite graph to obtain matching groups and isolated text elements; S4. Based on the page position relationship between the matching group and the isolated text element, reconstruct the globally unified reading order chain; S5. Based on the reading order chain, construct a set of constrained prompt words and candidate texts to form a large model's constrained input data; S6. Input the restricted input data into the large model to perform controlled optimization decision-making and output standardized text content; S7. Based on the standardized text content and full-link metadata, encapsulate and generate dual-type output results of structured data and readable text; Step S2 includes: taking all text elements output by the first OCR engine in the at least two OCR engines as the first side node set of the bipartite graph, taking all text elements output by the second OCR engine in the at least two OCR engines as the second side node set of the bipartite graph, and traversing all pairwise combinations of nodes in the first side node set and nodes in the second side node set to form candidate pairing relationships. For each candidate pairing, a normalized text similarity is calculated based on the edit distance between the text elements to be matched and the text length. The normalized text similarity includes calculations based on edit distance, n-gram similarity, character-level vector similarity, and field pattern similarity. The comprehensive matching weight is calculated based on the bounding box overlap and the normalized text similarity. For table areas or areas without valid text content, reduce or set the contribution of text similarity to the comprehensive matching weight; Step S2 further includes setting a lower bound threshold for bounding box overlap and a lower bound threshold for comprehensive matching weight, and retaining only candidate edges that simultaneously satisfy the lower bound threshold for bounding box overlap and the lower bound threshold for comprehensive matching weight; The feature weight coefficients and candidate edge filtering thresholds in the comprehensive matching weights are adaptively configured according to the document layout type, element type, or region type.

2. The method for multi-engine OCR result fusion and reading order optimization according to claim 1, characterized in that, Step S1 includes defining a single text element as having at least the following fields: text content, bounding box coordinates, page number, element type, source engine identifier, original sequence index, identification confidence, and page level identifier. The element type is also divided into at least one of the following: body text, title, table cell, header / footer, key-value pair field, and signature area.

3. The method for multi-engine OCR result fusion and reading order optimization according to claim 1, characterized in that, Step S3 includes: performing connected component retrieval on the weighted bipartite graph; performing community partitioning on connected subgraphs with more than a preset scale threshold using a preset community partitioning algorithm, which includes any one of Louvain's algorithm, Leiden's algorithm, spectral clustering, and threshold-based hierarchical clustering; determining the matching type (one-to-one, one-to-many, many-to-one, and many-to-many) based on the number of nodes corresponding to different OCR engines within each community, and calculating the matching quality score based on the comprehensive matching weight of candidate edges within the community; and marking nodes that have not formed a valid pair as isolated text elements.

4. The method for multi-engine OCR result fusion and reading order optimization according to claim 1, characterized in that, Step S4 includes: The elements on both sides are initially sorted according to their original index and page coordinates, and the matches are grouped and merged. Perform a two-pointer traversal based on the matching group merging results, and remap the traversal output to the final reading order index; Text elements in a regular layout are sorted according to the rule of prioritizing vertical coordinates and then horizontal coordinates; First, perform local matching and logical ordering of cells for the table area, and then incorporate the local table order into the global reading order chain; Isolated text elements are connected to the global reading order chain according to their relative positions in the original layout.

5. The method for multi-engine OCR result fusion and reading order optimization according to claim 1, characterized in that, Step S5 includes: limiting the large model to select text content only within the given candidate text set in the prompt words, performing only a small amount of error correction and formatting, outputting according to the reading order chain, and prohibiting autonomous generation outside the candidate text set; Step S6 includes: performing a comprehensive comparison and screening of the multiple candidate texts for each reading unit based on semantic integrity, character accuracy, and format standardization to determine the optimal benchmark text; performing redundant character removal, space normalization, punctuation correction, and content normalization on the optimal benchmark text; and not introducing new text content outside the scope of the candidate text set during the entire optimization process.

6. The method for multi-engine OCR result fusion and reading order optimization according to claim 1, characterized in that, Step S7 includes: generating structured JSON data including page number, reading order index, standardized text, bounding box, source engine identifier, matching type and confidence level; and synchronously generating sequential Markdown readable text according to the reading order chain; while retaining full-link traceability metadata for audit review.

7. A multi-engine OCR result fusion and reading order optimization system, characterized in that, It includes an input module, a parsing module, a graph matching module, a community segmentation module, a reading order module, a prompt word construction module, a large model optimization module, and an output module. These modules work together to execute the multi-engine OCR result fusion and reading order optimization method as described in any one of claims 1 to 6.

8. An electronic device, characterized in that, It includes a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, it implements the multi-engine OCR result fusion and reading order optimization method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the multi-engine OCR result fusion and reading order optimization method according to any one of claims 1 to 6.

Images