Document information extraction method and device, electronic equipment and storage medium

Abstract

The present disclosure relates to the technical field of information extraction, and discloses a single document information extraction method and device, electronic equipment and storage medium, which comprises the following steps: identifying a plurality of text boxes in a target single document image, and constructing an initial text box set according to the writing order; continuously iterating and executing a merging process, forming a candidate merging pair with any two adjacent text boxes in the text box set, obtaining multiple candidate merging pairs, respectively calculating the correlation scores of the two text boxes in each candidate merging pair in each correlation feature dimension, calculating the merging score of the candidate merging pair according to the correlation scores, screening out the candidate merging pair with the highest merging score and exceeding the preset score threshold as the target merging pair, splicing it into a new text box and updating the text box set, and stopping iteration until the merging scores of all candidate merging pairs do not exceed the threshold, taking the current text box set as the single document information extraction result.

Description

Technical Field

[0001] This disclosure relates to the technical field of information extraction, such as a method, apparatus, electronic device, and storage medium for extracting document information. Background Technology

[0002] In the field of document automation, automatically extracting key information from various documents such as invoices, contracts, application forms, and engineering drawings is a core step in improving business processing efficiency. Existing technologies primarily employ template matching and rule engines, OCR (Optical Character Recognition) combined with traditional machine learning, and single-modal deep learning models to extract information from these documents. Template matching and rule engines target documents with fixed layouts using preset templates or written rules to locate and extract information. OCR combined with traditional machine learning extracts text through optical character recognition and then uses natural language processing or traditional machine learning models for classification and extraction. Single-modal deep learning models utilize either vision-based convolutional neural networks for layout analysis or text-based Transformer models for semantic understanding. These existing technologies achieve a certain level of information extraction efficiency when processing standardized documents with uniform layouts and high printing quality.

[0003] However, in actual business scenarios, there are many complex situations with documents. Existing technologies have significant shortcomings when processing such complex documents, especially in the text information extraction stage, where there is a serious problem of text information fragmentation, which directly leads to a significant reduction in the completeness and accuracy of document information extraction.

[0004] For example, when performing text recognition on document images, OCR engines often segment logically continuous document text, such as a complete address, product name, or company name, into multiple fragmented and discrete text boxes to achieve higher recall rates. Existing technologies typically attempt to integrate these fragmented text boxes using simple merging rules based on geometric distance. This merging method is prone to errors, either incorrectly merging non-logically continuous text boxes or failing to effectively integrate logically continuous fragmented text boxes. This results in incomplete and logically confused extracted document information, failing to meet the actual business requirements for accurate and complete extraction of key document information. Summary of the Invention

[0005] To provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. This summary is not intended as a general commentary, nor is it intended to identify key / important components or describe the scope of protection of these embodiments, but rather as a prelude to the detailed description that follows.

[0006] This disclosure provides a method, apparatus, electronic device, and storage medium for extracting document information, which can achieve accurate and complete extraction of document information.

[0007] According to a first aspect of this disclosure, a method for extracting document information is provided, comprising: Text recognition is performed on the target document image to obtain multiple text boxes, and these text boxes are then constructed into an initial set of text boxes according to the order of the document content. The following merge process is executed iteratively: Form a candidate merge pair by combining any two adjacent text boxes in the current set of text boxes, and obtain multiple candidate merge pairs; Calculate the association scores of the two text boxes in each candidate merge pair on each dimension of the association features; The merging score of the candidate merging pair is calculated based on the association scores of the candidate merging pairs on each association feature dimension. The candidate merge pairs with the highest merge scores that exceed the preset score threshold are selected as the target merge pairs. The two text boxes in the target merge pairs are concatenated into a new text box, and the new text box set is updated. The iteration stops when the merging score of each candidate merging pair does not exceed the score threshold, and the current set of text boxes is taken as the result of document information extraction.

[0008] In some embodiments, the association feature includes spatial proximity, and the association score of the two text boxes in the candidate merge pair on the spatial proximity dimension is calculated in the following way: Calculate the geometric distance and alignment deviation between the two text boxes in the candidate merge pair; The geometric distance and alignment deviation are fused, and the result of the fusion is used as the association score of the two text boxes in the candidate merge pair in the spatial proximity dimension.

[0009] In some embodiments, the geometric distance between two text boxes includes at least one of the following: Vertical overlap ratio is calculated based on the intersection-union ratio of two text boxes in the vertical direction, and is used to reflect the degree of overlap between the two text boxes in the vertical direction. Horizontal spacing is calculated based on the ratio of the actual horizontal distance between the edges of two text boxes to the height of the text box characters, and is used to reflect the horizontal proximity of the two text boxes.

[0010] In some embodiments, the association feature includes semantic coherence, and the association score of the two text boxes in the candidate merge pair on the semantic coherence dimension is calculated as follows: Calculate the semantic relevance of the concatenated text boxes in the candidate merge pair; The semantic relevance is quantified, and the result of the quantification is used as the relevance score of the two text boxes in the candidate merging pair on the dimension of semantic coherence.

[0011] In some embodiments, the association feature includes layout consistency, and the association score of the two text boxes in the candidate merge pair on the layout consistency dimension is calculated in the following way: Calculate the similarity of various layout features of two text boxes in the candidate merge pair, wherein the layout features include at least one of font size, line height, indentation and color; The similarity of various layout features is fused, and the result of the fusion is used as the association score of the two text boxes in the candidate merge pair on the layout consistency dimension.

[0012] In some embodiments, the merging score of a candidate merging pair is calculated based on its association scores across various association feature dimensions, including: Determine the weights of the candidate merging pairs corresponding to their association scores on each association feature dimension; Based on each association score and its corresponding weight, a weighted fusion value of all association scores is calculated, and the weighted fusion value is used as the merging score of the candidate merging pair.

[0013] In some embodiments, prior to text recognition of the target document image, the method further includes: Calculate the scaling factor required to scale the original document image proportionally to fit it completely within the preset target size frame; The original document image is scaled proportionally according to the scaling factor; Zero-padding is performed on one side of the proportionally scaled document image until the document image reaches the preset target size, thus obtaining the target document image.

[0014] In some embodiments, calculating the scaling factor required to proportionally scale the original document image to fit completely within a preset target size frame includes: Determine the aspect ratio of the original document image; If the aspect ratio exceeds the preset aspect ratio threshold, the non-core area of ​​the original document image is cropped and adjusted to the preset reasonable aspect ratio range. The scaling factor required to scale the original document image after aspect ratio adjustment proportionally to fit completely into the preset target size frame is calculated. If the aspect ratio does not exceed the preset aspect ratio threshold, the scaling factor required to scale the original document image proportionally to fit it completely within the preset target size frame is directly calculated.

[0015] According to a second aspect of this disclosure, a document information extraction device is provided, comprising: The text box recognition module is configured to perform text recognition on the target document image, obtain multiple text boxes, and construct an initial text box set according to the order of the document content. The iterative execution module is configured to iteratively execute the following merging process: any two adjacent text boxes in the current text box set are combined into a candidate merging pair, resulting in multiple candidate merging pairs; the association scores of the two text boxes in each candidate merging pair are calculated on each association feature dimension; based on the association scores of the candidate merging pair on each association feature dimension, the merging score of the candidate merging pair is calculated; the candidate merging pair with the highest merging score exceeding a preset score threshold is selected as the target merging pair, the two text boxes in the target merging pair are concatenated into a new text box, and the new text box set is updated. The extraction result determination module is configured to stop iterating when the merging score of each candidate merging pair does not exceed the score threshold, and to use the current set of text boxes as the document information extraction result.

[0016] According to a third aspect of this disclosure, an electronic device is provided, including a processor and a memory storing computer program instructions, the processor being configured to execute the document information extraction method provided in the first aspect of this disclosure when running the program instructions.

[0017] According to a fourth aspect of this disclosure, a storage medium is provided that stores computer program instructions, which, when executed by a processor, perform the document information extraction method provided in the first aspect of this disclosure.

[0018] The document information extraction method, apparatus, electronic device, and storage medium provided in this disclosure can achieve the following technical effects: After text recognition is performed on the target document image, an initial set of text boxes is constructed according to the order of the document content, ensuring that the text box processing aligns with the content logic of the document itself. Then, an iterative merging process is performed to accurately stitch together the fragmented text boxes. During the iteration process, adjacent text boxes in the current text box set are first formed into candidate merging pairs. Then, the merging score is obtained by calculating the correlation score of each candidate merging pair in various correlation feature dimensions, achieving a comprehensive consideration of the multi-dimensional correlation between text boxes. The candidate merging pair with the highest merging score that exceeds a preset score threshold is then selected as the target merging pair for stitching and updating the text box set. Through this iterative merging strategy, fragmented text boxes are gradually and accurately integrated. At the same time, the preset score threshold serves as a clear merging criterion to avoid meaningless merging of low-correlation text boxes. The iteration stops when there are no candidate merging pairs that meet the conditions. The final output text box set can effectively integrate the originally fragmented and discrete text boxes into complete text boxes that can clearly express semantics, achieving accurate and complete extraction of document information.

[0019] The above general description and the description below are exemplary and illustrative only and are not intended to limit this disclosure. Attached Figure Description

[0020] One or more embodiments are illustrated by way of example with reference to the accompanying drawings. These illustrations and drawings do not constitute a limitation on the embodiments. Elements having the same reference numerals in the drawings are shown as similar elements. The drawings are not to be scaled. And wherein: Figure 1 This is a flowchart illustrating a document information extraction method provided in an embodiment of this disclosure; Figure 2 This is a flowchart illustrating another document information extraction method provided in this embodiment of the disclosure; Figure 3 This is a flowchart illustrating another document information extraction method provided in this embodiment of the disclosure; Figure 4 This is a schematic diagram of a document information extraction device provided in an embodiment of this disclosure; Figure 5 This is a schematic diagram of another document information extraction device provided in this embodiment of the present disclosure; Figure 6 This is a schematic diagram of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0021] To provide a more detailed understanding of the features and technical content of the embodiments of this disclosure, the implementation of the embodiments of this disclosure will be described in detail below with reference to the accompanying drawings. The accompanying drawings are for illustrative purposes only and are not intended to limit the embodiments of this disclosure. In the following technical description, for ease of explanation, several details are used to provide a full understanding of the disclosed embodiments. However, one or more embodiments may still be implemented without these details. In other cases, well-known structures and devices may be simplified in their depiction to simplify the drawings.

[0022] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this disclosure described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion.

[0023] Unless otherwise stated, the term "multiple" means two or more.

[0024] In this embodiment of the disclosure, the character " / " indicates that the objects before and after it are in an "or" relationship. For example, A / B means: A or B.

[0025] The term "and / or" describes an association between objects, indicating that three relationships can exist. For example, A and / or B means: A or B, or A and B.

[0026] The term "correspondence" can refer to an association or binding relationship. The correspondence between A and B means that there is an association or binding relationship between A and B.

[0027] In the field of document automation, automatically extracting key information from various documents such as invoices, contracts, application forms, and engineering drawings is a core step in improving business processing efficiency. Existing technologies primarily employ template matching and rule engines, OCR combined with traditional machine learning, and single-modal deep learning models to extract information from these documents. Template matching and rule engines target documents with fixed layouts using preset templates or written rules to locate and extract information. OCR combined with traditional machine learning extracts text through optical character recognition and then uses natural language processing or traditional machine learning models for classification and extraction. Single-modal deep learning models utilize either vision-based convolutional neural networks for layout analysis or text-based Transformer models for semantic understanding. These existing technologies can achieve a certain level of information extraction efficiency when processing standardized documents with uniform layouts and high print quality.

[0028] However, in actual business scenarios, there are many complex situations with documents. Existing technologies have significant shortcomings when processing such complex documents, especially in the text information extraction stage, where there is a serious problem of text information fragmentation, which directly leads to a significant reduction in the completeness and accuracy of document information extraction.

[0029] For example, when performing text recognition on document images, OCR engines often segment logically continuous document text, such as a complete address, product name, or company name, into multiple fragmented and discrete text boxes to achieve higher recall rates. Existing technologies typically attempt to integrate these fragmented text boxes using simple merging rules based on geometric distance. This merging method is prone to errors, either incorrectly merging non-logically continuous text boxes or failing to effectively integrate logically continuous fragmented text boxes. This results in incomplete and logically confused extracted document information, failing to meet the actual business requirements for accurate and complete extraction of key document information.

[0030] Furthermore, existing technologies have extremely poor generalization ability for complex document layouts. Template and rule-based methods lack generalization ability and immediately fail when the layout changes to multiple columns, spanning pages, nested tables, or interlaced rows and columns. Deep learning models trained from scratch also struggle to adapt to unfamiliar new layouts. They also lack robustness to low-quality document images. Problems such as rotation, perspective distortion, seal occlusion, handwritten annotations, image blurring, and uneven lighting that occur during document circulation and archiving are severely amplified due to the limitations of traditional image preprocessing methods, interfering with subsequent recognition. Simultaneously, the independent visual and textual models are fragmented, with image and text information existing in isolation and unable to be effectively integrated. The model struggles to understand the deep layout and semantic relationships of the document, further affecting the accuracy of information extraction.

[0031] This disclosure provides an electronic device, which is a device with computing processing capabilities, specifically including computers, terminal devices, and servers. In conjunction with the electronic device provided in this disclosure (hereinafter referred to as the device), this disclosure provides a method for extracting document information, such as... Figure 1 As shown, the methods for extracting document information include: S101, the device performs text recognition on the target document image, obtains multiple text boxes, and constructs an initial set of text boxes according to the order of the document content.

[0032] In this embodiment, a text box is an independent recognition unit obtained after text recognition of the target document image. Specifically, it is a combination of characters with spatial continuity in the target document image, such as a single word, a line of text, or a field value. After obtaining multiple text boxes generated by the recognition of the target document image, based on the spatial coordinate information of each text box, and combined with the inherent natural text order and visual layout logic of the document (such as from left to right, from top to bottom, row and column arrangement for table-type documents, and hierarchical progression for list-type documents), all discrete text boxes are sorted and arranged in an orderly manner. Following the actual expression order of the information carried by the document, multiple text boxes are integrated into an ordered initial set of text boxes. This lays a logical foundation that fits the actual business scenario for subsequent processing steps such as filtering and merging of adjacent text boxes, ensuring the rationality and accuracy of subsequent fragmented text box integration operations.

[0033] In this embodiment of the disclosure, the operations of S102 to S105 are integrated into a merging process. After the operation of S101 is completed, the merging process is executed iteratively.

[0034] S102, the device forms a candidate merge pair by any two adjacent text boxes in the current set of text boxes, and obtains multiple candidate merge pairs.

[0035] In this embodiment, after constructing the initial set of text boxes, a candidate merging pair construction operation is performed on the current ordered set of text boxes. Specifically, the current set of text boxes is traversed, and each pair of adjacent text boxes in each order is combined to form a candidate merging pair. This traversal and combination method can cover all text box combinations in the current set that meet the logical adjacency relationship, ultimately obtaining multiple candidate merging pairs corresponding to the number of adjacent text box groups. For example, if the current set of text boxes contains text boxes A, B, C, and D in the order of text, then A and B, B and C, and C and D are respectively formed into candidate merging pairs, ultimately obtaining three sets of candidate merging pairs: (A, B), (B, C), and (C, D). This operation can lock all text box combinations with logical merging potential, providing a clear processing object for subsequent calculation of the association score of each combination and screening of target merging pairs, while also avoiding including non-logically consecutive text boxes in the candidate range.

[0036] S103, the device calculates the association scores of the two text boxes in each candidate merge pair on each dimension of the association features.

[0037] In this embodiment, multiple association features can be configured. These features characterize the degree of association between two text boxes, and each feature is matched with a corresponding quantization calculation rule. For the two text boxes in a single candidate merge pair, the device will sequentially perform feature extraction and numerical calculation according to the quantization calculation rules of each association feature dimension to obtain the quantization score of the candidate merge pair under each association feature dimension. This quantization score is the association score for the corresponding dimension. The association score for each dimension is a numerical value that quantifies the degree of association between the two text boxes on the corresponding feature dimension. The higher the score, the stronger the association between the two on that dimension.

[0038] S104, the device calculates the merging score of the candidate merging pair based on the association scores of the candidate merging pairs on each association feature dimension.

[0039] In this embodiment of the disclosure, the association scores of candidate merging pairs on each association feature dimension can be fused, and the fused result can be used as the merging score of the candidate merging pair. For example, the association scores can be fused to obtain the corresponding merging score by performing operations such as averaging or summing on the association scores of each association feature dimension.

[0040] S105, the device selects the candidate merging pair with the highest merging score that exceeds the preset score threshold as the target merging pair, concatenates the two text boxes in the target merging pair into a new text box, and updates the new text box set.

[0041] In this embodiment, the score threshold is a critical value for determining the rationality of text box merging, and is determined comprehensively based on the business scenario requirements of document information extraction, the characteristics of actual document data, and the model training and verification results. For example, based on historical data and business experience of massive documents, an initial benchmark threshold adapted to common scenarios is preset; or, for specific industries and types of documents, a specific critical threshold is determined by analyzing the merging scores of labeled positive and negative samples during sample training; or, the score threshold is dynamically fine-tuned according to indicators such as the accuracy and completeness of the actually extracted text box merging, increasing the score threshold if there are many incorrect mergings and decreasing the score threshold if there are many missed mergings, thus ensuring the merging effect through closed-loop optimization.

[0042] In this embodiment, the merging scores of all current candidate merging pairs are globally statistically analyzed and sorted. The candidate merging pair with the highest merging score is selected, and its merging score is compared with a preset score threshold. If the highest merging score exceeds the preset score threshold, the candidate merging pair is determined as the target merging pair. Then, the text content of the two text boxes in the target merging pair is concatenated to generate a new text box. Finally, the current text box set is updated, the two text boxes in the original target merging pair are deleted, and the newly generated text box is included in the text box set according to the original text order, forming a new text box set after the completion of this round of merging, providing a new processing object for the next round of iteration.

[0043] S106, the device stops iterating when the merging score of each candidate merging pair does not exceed the score threshold, and uses the current set of text boxes as the result of document information extraction.

[0044] When the merging scores of all candidate merging pairs in the current text box set do not exceed the score threshold, it indicates that there are no text box combinations in the current text box set that have logical continuity and are suitable for merging. At this time, the iterative merging process will be terminated and no further merging operations will be performed. The text box set in the current state will be output as the final document information extraction result. Each text box in this set is a semantically complete and logically independent information unit.

[0045] The document information extraction method provided in this embodiment first performs text recognition on the target document image, then constructs an initial set of text boxes according to the text order of the document content, ensuring that the text box processing aligns with the content logic of the document itself. Then, iterative merging processing is performed to accurately stitch together fragmented text boxes. During the iteration process, adjacent text boxes in the current set are first considered as candidate merging pairs. Then, a merging score is obtained by calculating the correlation scores of each candidate merging pair across various correlation feature dimensions, achieving a comprehensive consideration of the multi-dimensional correlation between text boxes. The candidate merging pair with the highest merging score exceeding a preset score threshold is selected as the target merging pair for stitching and updating the text box set. Through this iterative merging strategy, fragmented text boxes are progressively and accurately integrated. Simultaneously, a preset score threshold serves as a clear merging criterion, avoiding meaningless merging of low-correlation text boxes. Iteration stops when no candidate merging pairs meet the criteria. The final output set of text boxes effectively integrates the originally fragmented and discrete text boxes into complete text boxes that clearly express semantics, achieving accurate and complete extraction of document information.

[0046] In some embodiments, the association feature includes spatial proximity, which quantifies the closeness of two text boxes in terms of spatial location and typographic alignment. The association score of the two text boxes in the candidate merge pair in the spatial proximity dimension is calculated as follows: the geometric distance and alignment deviation of the two text boxes in the candidate merge pair are calculated; the geometric distance and alignment deviation are fused, and the result of the fusion is used as the association score of the two text boxes in the candidate merge pair in the spatial proximity dimension. Specifically, the geometric distance and alignment deviation can be weighted or normalized, and the fused comprehensive value is used as the association score of the candidate merge pair in the spatial proximity dimension.

[0047] In this embodiment of the disclosure, the alignment deviation of the two text boxes is used to quantify the degree of alignment regularity of the two text boxes in the layout direction, specifically by calculating the difference between the coordinates of the left edge or the center point of the two text boxes.

[0048] In this embodiment of the disclosure, the geometric distance between two text boxes includes a vertical overlap ratio, which is calculated based on the intersection-union ratio (IoU) of the two text boxes in the vertical direction and is used to reflect the degree of overlap between the two text boxes in the vertical direction. Specifically, the vertical overlap ratio is calculated using the Y-axis IoU (vertical intersection-union ratio), which is obtained by calculating the ratio of the overlapping area of ​​the two text boxes in the vertical direction to the overall area.

[0049] In this embodiment of the disclosure, the geometric distance between two text boxes includes a horizontal spacing, which is calculated based on the ratio of the actual horizontal distance between the edges of the two text boxes to the character height of the text boxes, and is used to reflect the proximity of the two text boxes in the horizontal direction. Specifically, the actual horizontal distance between the edges of the two text boxes is first calculated, and then the ratio of the actual horizontal distance to the character height of the text boxes is calculated. The final ratio is the horizontal spacing.

[0050] This embodiment of the disclosure decomposes spatial proximity into geometric distance (vertical overlap rate, horizontal spacing) and alignment deviation for joint calculation, and fuses multi-dimensional indicators to quantitatively evaluate the spatial relationship between two text boxes from three aspects: vertical belonging, horizontal proximity, and layout alignment regularity. The relationship score obtained after fusing multiple indicators can more accurately determine whether two text boxes are the same continuous text, significantly improving the accuracy and rationality of text box merging.

[0051] In some embodiments, the association feature includes semantic coherence, which is used to quantify the degree of semantic association between two text boxes from the perspectives of text content, semantic logic, and language expression habits. The association score of the two text boxes in the candidate merge pair on the semantic coherence dimension is calculated as follows: calculate the semantic association degree after splicing the two text boxes in the candidate merge pair; quantify the semantic association degree, and use the result of the quantification as the association score of the two text boxes in the candidate merge pair on the semantic coherence dimension.

[0052] In this embodiment of the disclosure, a lightweight text vector model can be used to perform semantic encoding and similarity calculation on the text content of a single text box and the text content of the concatenated two text boxes, respectively, to determine whether the semantics of the concatenated text is more coherent and complete than that of a single text box, thereby obtaining the semantic relevance of the concatenated two text boxes in the candidate merging pair; the semantic relevance is then quantified, and the quantified value is used as the relevance score of the two text boxes in the candidate merging pair on the dimension of semantic coherence.

[0053] Lightweight text vector models (TVMs) are text semantic encoding models with small parameter counts, low computational overhead, and fast inference speed, suitable for edge or real-time computing scenarios. They map text content into fixed-dimensional semantic vectors to achieve quantitative calculations of text semantic similarity and coherence. Lightweight TVMs are typically implemented based on lightweight architectures of pre-trained language models, such as using distillation, quantization, pruning, or lightweight network structures like DistilBERT, TinyBERT, lightweight Word2Vec, and lightweight Sentence-BERT. Here, lightweight TVMs are used to semantically encode the content of a single text box and the content of two concatenated text boxes. By calculating the semantic vector similarity or semantic completeness score before and after concatenation, they determine whether the concatenated text is more fluent and logically coherent, thus achieving a quantitative score for semantic coherence.

[0054] In some embodiments, the association feature includes layout consistency. Layout consistency is used to quantify the similarity of layout attributes between two text boxes at the level of typography styles such as font size, line height, indentation, and color. The association score of the two text boxes in the candidate merge pair on the layout consistency dimension is calculated as follows: the similarity of each layout feature of the two text boxes in the candidate merge pair is calculated, wherein the layout feature includes at least one of font size, line height, indentation, and color; the similarity of each layout feature is fused, and the result of the fusion is used as the association score of the two text boxes in the candidate merge pair on the layout consistency dimension.

[0055] In this embodiment of the disclosure, multiple layout features corresponding to the two text boxes in the candidate merge pair can be extracted respectively. The layout features include at least one of font size, line height, indentation, and color. For each layout feature, the similarity between the two text boxes on the layout feature is calculated to obtain the similarity score of the single layout feature. Then, the similarity scores of each layout feature are weighted, normalized, or directly summed and fused. The fused comprehensive value is used as the association score of the candidate merge pair on the layout consistency dimension.

[0056] In some embodiments, the merging score of a candidate merging pair can be calculated based on its association scores in the dimensions of spatial proximity, semantic coherence, and layout consistency. By comprehensively considering the association scores across these three dimensions, the merging score can be obtained, allowing for a comprehensive assessment of whether text boxes belong to the same continuous content segment from multiple perspectives, including spatial location, semantic logic, and layout style. This avoids erroneous or incomplete merging due to single-dimensional judgments, improving the accuracy and stability of text box merging and enhancing the overall precision of document information extraction.

[0057] In some embodiments, the merging score of a candidate merging pair is calculated based on its correlation scores across various correlation feature dimensions. This includes: determining the weights corresponding to the correlation scores of the candidate merging pair across various correlation feature dimensions; calculating a weighted fusion value of all correlation scores based on each correlation score and its corresponding weight; and using the weighted fusion value as the merging score of the candidate merging pair. Here, weights can be assigned to each correlation feature according to document type, layout characteristics, and business experience.

[0058] By assigning different weights to various related features, the proportion of each related feature dimension in the consolidation decision can be flexibly adjusted according to the specific type of document (such as invoices, contracts, and engineering drawings) and its layout characteristics. For example, for invoices with layout rules, the weight of spatial proximity and layout consistency can be increased; for contracts with complex content and heavy semantic logic, the weight of semantic coherence can be increased. This makes the consolidation score calculation highly consistent with actual business scenarios and significantly improves the accuracy of consolidation judgment.

[0059] Taking the calculation of the merge score of a candidate merge pair based on its association scores in the dimensions of spatial proximity, semantic coherence, and layout consistency as an example, the expression for the calculation process of the merge score is as follows: Score = w1 f(spatial)+w2 g(semantic)+w3 h(layout).

[0060] Where f(spatial) is the association score of the candidate merge pair in the spatial proximity dimension, g(semantic) is the association score of the candidate merge pair in the semantic coherence dimension, and h(layout) is the association score of the candidate merge pair in the layout consistency dimension; w1, w2 and w3 are the weights of the association scores of the candidate merge pair in the spatial proximity, semantic coherence and layout consistency dimensions, respectively.

[0061] This disclosure provides another method for extracting document information, such as... Figure 2 As shown, the methods for extracting document information include: S201, the device performs text recognition on the target document image, obtains multiple text boxes, and constructs an initial set of text boxes according to the order of the document content.

[0062] In this embodiment of the disclosure, the operations of S202 to S206 are integrated into a merge process. After the operation of S201 is completed, the merge process is executed iteratively.

[0063] S202, the device forms a candidate merge pair by any two adjacent text boxes in the current set of text boxes, and obtains multiple candidate merge pairs.

[0064] S203, the device calculates the association scores of the two text boxes in each candidate merge pair on each dimension of the association features.

[0065] S204, the device determines the weights corresponding to the association scores of the candidate merging pairs on each association feature dimension.

[0066] S205, the device calculates the weighted fusion value of all associated scores based on each associated score and its corresponding weight, and uses the weighted fusion value as the merging score of the candidate merging pair.

[0067] S206, the device selects the candidate merge pairs with the highest merge score that exceeds the preset score threshold as the target merge pairs, concatenates the two text boxes in the target merge pairs into a new text box, and updates the new text box set.

[0068] S207, the device stops iterating when the merging score of each candidate merging pair does not exceed the score threshold, and uses the current set of text boxes as the result of document information extraction.

[0069] In some embodiments, before performing text recognition on the target document image, the method further includes: calculating a scaling factor required to proportionally scale the original document image to fit completely within a preset target size frame; proportionally scaling the original document image according to the scaling factor; and padding one side of the proportionally scaled document image with zeros until the size of the document image reaches the preset target size, thereby obtaining the target document image.

[0070] Based on the length and width of the original document image and the preset target size, a scaling factor is calculated to ensure that the original document image can be scaled proportionally and completely placed within the preset target size frame. The original document image is then scaled proportionally according to this scaling factor, ensuring that the image is not stretched, compressed, or distorted. Zero-padding is performed on one side (such as the right side and bottom) of the proportionally scaled document image until the image size reaches the preset target size, resulting in a uniformly sized target document image. This method can unify document images of any size to a fixed preset target size without distorting, stretching, or compressing the image content. It completely avoids the information loss and image distortion caused by traditional scaling methods, preserving the original form, proportion, and relative position of text, tables, layout, and other content in the image, which is beneficial for obtaining accurate text boxes in subsequent text recognition.

[0071] This disclosure provides another method for extracting document information, such as... Figure 3 As shown, the methods for extracting document information include: S301, the device calculates the scaling factor required to proportionally scale the original document image to fit completely within the preset target size frame.

[0072] S302, the device scales the original document image proportionally according to the scaling factor.

[0073] S303, the device performs zero-padding on one side of the proportionally scaled document image until the document image size reaches the preset target size, thus obtaining the target document image.

[0074] S304, the device performs text recognition on the target document image, obtains multiple text boxes, and constructs an initial set of text boxes according to the order of the document content.

[0075] In this embodiment of the disclosure, the operations of S305 to S308 are integrated into a merge process. After the operation of S304 is completed, the merge process is executed iteratively.

[0076] S305, the device forms a candidate merge pair by any two adjacent text boxes in the current set of text boxes, and obtains multiple candidate merge pairs.

[0077] S306, the device calculates the association scores of the two text boxes in each candidate merge pair on each association feature dimension.

[0078] S307, the device calculates the merging score of the candidate merging pair based on the association scores of the candidate merging pairs on each association feature dimension.

[0079] S308, the device selects the candidate merge pairs with the highest merge score that exceeds the preset score threshold as the target merge pairs, concatenates the two text boxes in the target merge pairs into a new text box, and updates the new set of text boxes.

[0080] S309, the device stops iterating when the merging score of each candidate merging pair does not exceed the score threshold, and uses the current set of text boxes as the result of document information extraction.

[0081] In some embodiments, calculating the scaling factor required to proportionally scale the original document image to fit completely within a preset target size frame includes: determining the aspect ratio of the original document image; if the aspect ratio exceeds a preset aspect ratio threshold, cropping the non-core area of ​​the original document image and adjusting it to a preset reasonable aspect ratio range, and calculating the scaling factor required to proportionally scale the original document image after aspect ratio adjustment to fit completely within the preset target size frame; if the aspect ratio does not exceed the preset aspect ratio threshold, directly calculating the scaling factor required to proportionally scale the original document image to fit completely within the preset target size frame.

[0082] When the aspect ratio of the original document image exceeds a preset threshold, it indicates that the original document image is an extreme proportion image (such as an oversized receipt or an oversized drawing). In this case, the non-core areas of the original document image (such as blank areas, headers, footers, and redundant areas at the ends) are appropriately cropped to adjust the aspect ratio to a reasonable range. This operation significantly reduces invalid fill areas, avoids wasting computational resources due to excessively large blank areas, and achieves an optimal balance between preserving core business information and controlling computational overhead. Simultaneously, without distorting, stretching, or compressing the image content, it unifies document images of any size to a fixed preset target size, completely avoiding information loss and image distortion caused by traditional scaling methods. This ensures that the original form, proportion, and relative position of text, tables, layout, and other content in the image are fully preserved, which is beneficial for obtaining accurate text boxes in subsequent text recognition. It is particularly suitable for the efficient and stable processing of oversized and oversized irregularly shaped document images.

[0083] In this embodiment, a multimodal pre-trained model is introduced. Here, a multimodal pre-trained model refers to a deep learning model that is pre-trained on large-scale multimodal data and can simultaneously encode and uniformly represent information from at least two modalities among text modality, image visual modality, and layout modality. For example, the multimodal pre-trained model can be a multimodal image-text model such as LayoutLMv2 (Multi-modal Pre-training for Visually-rich Document Understanding) or DiT (DiffusionTransformer). The introduced multimodal pre-trained model is used as a feature extractor. This multimodal pre-trained model is pre-trained on billions of general image-text pairs (such as web pages, images with captions, academic papers, etc.) and possesses rich general knowledge and prior capabilities, specifically including visual semantic understanding capabilities and layout structure understanding capabilities. The multimodal pre-trained model can recognize visual elements such as tables, seals, handwritten signatures, and logos; it can also understand general layout rules such as text arrangement order, the relationship between titles and body text, and list hierarchy structure.

[0084] By fine-tuning the multimodal pre-trained model on a small scale of specific business document data, a dedicated recognition model adapted to document image recognition tasks is obtained. This allows the general visual, text, and layout knowledge learned during pre-training to be quickly transferred to the target business domain, significantly reducing the amount of labeled data required for model training and effectively solving the technical problems of large data dependence, weak generalization ability, and difficulty in cold start when training a model from scratch.

[0085] In this embodiment, text recognition of a target document image to obtain multiple text boxes can be achieved as follows: The target document image is input into a dedicated recognition model for global analysis, outputting text region, layout structure, and visual element information to guide the OCR module in text recognition, resulting in preliminary text boxes. The dedicated recognition model then corrects the position of the preliminary text boxes, removes duplicates, and filters them, optimizing the text box coordinates and filtering invalid text boxes to obtain multiple text boxes containing text content and spatial coordinates. Specifically, the target document image is input into a dedicated recognition model, which performs global analysis on the document image, outputting text region positioning results, layout structure information, and visual element differentiation results. This determines the text regions, text arrangement order, and the positions of visual elements such as tables, seals, and handwriting in the image, providing accurate text region guidance for subsequent OCR. Based on the text region guidance results output by the dedicated recognition model, the OCR module performs optical character recognition on the text regions of the target document image, detecting and locating the text positions region by region to obtain preliminary text boxes and their corresponding text content. A dedicated recognition model corrects the position, removes duplicates, and filters the initial text boxes output by OCR. Combining the text arrangement order, title-body relationship, list hierarchy and other layout rules, it filters invalid text boxes and optimizes the coordinates of the text boxes, finally obtaining multiple text boxes with accurate positions that conform to the layout logic. Each text box contains text content and corresponding spatial coordinate information.

[0086] The following describes in detail the specific implementation process of the document information extraction method provided in this embodiment, using three document information extraction scenarios as examples.

[0087] Scenario 1: Information extraction from tilted invoices Using a 15° tilted invoice image as the original document image, when processing the original document image with a 15° tilt, the angle of the original document image is first corrected by affine transformation; then the scaling factor required to proportionally scale the corrected original document image to fit completely within the preset target size frame is calculated, and the original document image is proportionally scaled according to the scaling factor, and zero-padding is performed on one side of the scaled original document image until the size of the original document image reaches the preset target size, resulting in a target document image without content distortion.

[0088] The target document image is input into a dedicated recognition model, which performs a global analysis of the image, outputting text region location results, layout structure information, and visual element differentiation results. This determines the text regions, text arrangement order, and the positions of visual elements such as printed and handwritten text, providing accurate text region guidance for subsequent OCR. Based on the guidance results output by the dedicated recognition model, the OCR module performs optical character recognition on the text regions, detecting and locating the text positions region by region to obtain preliminary text boxes. The dedicated recognition model then performs position correction, deduplication, and filtering on the preliminary text boxes, resulting in multiple text boxes with accurate positions that conform to the layout logic, including three discrete text boxes: "Jiangsu Province," "Nanjing City," and "XX Road."

[0089] The above multiple text boxes are constructed into an initial text box set according to the order of the invoice content, and the merging process is iteratively executed: any two adjacent text boxes in the current text box set are formed into candidate merging pairs, and the association scores of each candidate merging pair in terms of spatial proximity, semantic coherence, and layout consistency are calculated; based on each association score and its corresponding weight, a weighted fusion value is calculated as the merging score; the candidate merging pair with the highest merging score and exceeding the preset score threshold is selected as the target merging pair, the two text boxes in the target merging pair are concatenated into a new text box, and the new text box set is updated.

[0090] During the iteration process, "Jiangsu Province", "Nanjing City", and "XX Road" are merged into a single complete address field because they are spatially adjacent, have the same layout, and all describe addresses. Their merge scores meet the merging criteria. The iteration stops when the merge scores of all candidate merge pairs do not exceed the score threshold, and the current set of text boxes is used as the document information extraction result, achieving accurate information extraction from tilted invoices.

[0091] Scenario 2: Extracting information from the product name on the newline of the purchase order The purchase order image with line breaks for product names is used as the original document image. A scaling factor is calculated to proportionally scale the original document image to fit completely within a preset target size frame. The image is then proportionally scaled according to the scaling factor, with zero padding applied to one side of the scaled original document image until the image size reaches the preset target size, resulting in a target document image without content distortion. This target document image is then input into a dedicated recognition model, which performs global analysis and outputs information on text regions, layout structure, and visual elements to guide the OCR module in performing optical character recognition, obtaining preliminary text boxes. The dedicated recognition model then corrects the position of these preliminary text boxes, removes duplicates, and filters them, resulting in multiple text boxes containing text content and spatial coordinates.

[0092] Due to line breaks in the layout, the product name "High-performance industrial-grade laser printer (including spare ink cartridges)" was recognized by OCR as two separate text boxes: "High-performance industrial-grade laser printer" and "(including spare ink cartridges)". All text boxes were arranged in the order of the text to form an initial set of text boxes, and a merging process was iteratively performed: adjacent text boxes in the current set were considered candidate merge pairs, and the association scores for each candidate merge pair were calculated in terms of spatial proximity, semantic coherence, and layout consistency. Specifically, the two text boxes were vertically adjacent and left-aligned, resulting in a high spatial proximity score; they also had the same font and size, leading to a high layout consistency score; and the merged text had more complete semantics, resulting in a high semantic coherence score.

[0093] The weighted fusion value is calculated based on the correlation scores and their corresponding weights to obtain the merging score of the candidate merging pair. The pair with the highest merging score exceeding a preset threshold is identified as the target merging pair. The two text boxes are then concatenated into a complete product name text box, and the text box set is updated. Iteration stops when no candidate merging pair meets the criteria. The current text box set is used as the final result for extracting document information, avoiding information extraction errors caused by line breaks.

[0094] Scenario 3: Extracting information from outbound orders with extreme aspect ratios and obscured by stamps The image of a long, thin outbound slip with an aspect ratio far exceeding the norm and the quantity field obscured by a red "Outbound" stamp is used as the original document image. First, the aspect ratio of the original document image is determined to exceed the preset aspect ratio threshold. Then, non-core areas such as the signature bar at the bottom of the image and a large amount of blank space are intelligently cropped to adjust the aspect ratio to a preset reasonable range. Next, the scaling factor required to proportionally scale the image after aspect ratio adjustment to fit completely within the preset target size frame is calculated. The image is scaled proportionally according to the scaling factor, and zero-padding is performed on one side of the scaled image until the preset target size is reached, resulting in a standard-sized target document image that retains core information and has no content distortion.

[0095] In the text recognition stage, the target document image is input into a dedicated recognition model. This model performs a global analysis of the image, identifying the table layout structure and visual elements of the seal, and inferring the text attributes of areas obscured by the seal, providing accurate text region guidance for OCR. Based on the guidance results, the OCR module performs optical character recognition to obtain preliminary text boxes. The dedicated recognition model further corrects the position of these preliminary text boxes, removes duplicates, and filters them, resulting in multiple text boxes that conform to the layout logic. Specifically, "250ml24 bottles / box" in the specifications column is recognized as two separate text boxes: "250ml24" and "bottle / box".

[0096] During the merging iteration phase, all text boxes are constructed into an initial set of text boxes in the order of their text, and the merging process is performed iteratively: adjacent text boxes are formed into candidate merge pairs, and the association scores of each candidate merge pair in terms of spatial proximity, semantic coherence, and layout consistency are calculated. The two text boxes mentioned above are located within the same table cell, are spatially adjacent, have consistent layout characteristics, and, when concatenated, form a complete product specification semantic. Their merging scores meet the merging conditions, and they are concatenated into a complete specification field.

[0097] When the merging scores of each candidate merging pair do not exceed the score threshold, the iteration stops, and the current set of text boxes is taken as the result of document information extraction. Even in complex scenarios with extreme aspect ratios and seal obscuration, complete and accurate information extraction can still be achieved.

[0098] Combination Figure 4 As shown, this embodiment of the present disclosure provides a document information extraction device 400, which includes a text box recognition module 401, an iterative execution module 402, and an extraction result determination module 403.

[0099] The text box recognition module 401 is configured to: perform text recognition on the target document image to obtain multiple text boxes, and construct an initial text box set by arranging the multiple text boxes according to the order of the document content.

[0100] The iterative execution module 402 is configured to iteratively execute the following merging process: form a candidate merging pair from any two adjacent text boxes in the current text box set, and obtain multiple candidate merging pairs; calculate the association scores of the two text boxes in each candidate merging pair on each association feature dimension; calculate the merging score of the candidate merging pair based on the association scores of the candidate merging pair on each association feature dimension; select the candidate merging pair with the highest merging score that exceeds the preset score threshold as the target merging pair, concatenate the two text boxes in the target merging pair into a new text box, and update the new text box set.

[0101] The extraction result determination module 403 is configured to stop iterating when the merging score of each candidate merging pair does not exceed the score threshold, and to take the current set of text boxes as the document information extraction result.

[0102] The document information extraction device 400 provided in this embodiment first performs text recognition on the target document image, then constructs an initial set of text boxes according to the order of the document content, ensuring that the text box processing conforms to the content logic of the document itself. Then, iterative merging processing is performed to accurately stitch together fragmented text boxes. During the iteration process, adjacent text boxes in the current text box set are first considered as candidate merging pairs. Then, a merging score is obtained by calculating the correlation scores of each candidate merging pair across various correlation feature dimensions, achieving a comprehensive consideration of the multi-dimensional correlation between text boxes. The candidate merging pair with the highest merging score exceeding a preset score threshold is then selected as the target merging pair for stitching and updating the text box set. Through this iterative merging strategy, fragmented text boxes are progressively and accurately integrated. Simultaneously, a preset score threshold serves as a clear merging criterion, avoiding meaningless merging of low-correlation text boxes. Iteration stops when no candidate merging pairs meet the conditions. The final output text box set effectively integrates the originally fragmented and discrete text boxes into complete text boxes that clearly express semantics, achieving accurate and complete extraction of document information.

[0103] In some embodiments, the association feature includes spatial proximity, and the association score of the two text boxes in the candidate merge pair on the spatial proximity dimension is calculated in the following way: Calculate the geometric distance and alignment deviation between the two text boxes in the candidate merge pair; The geometric distance and alignment deviation are fused, and the result of the fusion is used as the association score of the two text boxes in the candidate merge pair in the spatial proximity dimension.

[0104] In some embodiments, the geometric distance between two text boxes includes at least one of the following: Vertical overlap ratio is calculated based on the intersection-union ratio of two text boxes in the vertical direction, and is used to reflect the degree of overlap between the two text boxes in the vertical direction. Horizontal spacing is calculated based on the ratio of the actual horizontal distance between the edges of two text boxes to the height of the text box characters, and is used to reflect the horizontal proximity of the two text boxes.

[0105] In some embodiments, the association feature includes semantic coherence, and the association score of the two text boxes in the candidate merge pair on the semantic coherence dimension is calculated as follows: Calculate the semantic relevance of the concatenated text boxes in the candidate merge pair; The semantic relevance is quantified, and the result of the quantification is used as the relevance score of the two text boxes in the candidate merging pair on the dimension of semantic coherence.

[0106] In some embodiments, the association feature includes layout consistency, and the association score of the two text boxes in the candidate merge pair on the layout consistency dimension is calculated in the following way: Calculate the similarity of various layout features of two text boxes in the candidate merge pair, wherein the layout features include at least one of font size, line height, indentation and color; The similarity of various layout features is fused, and the result of the fusion is used as the association score of the two text boxes in the candidate merge pair on the layout consistency dimension.

[0107] In some embodiments, the iteration execution module 402 is configured to: Determine the weights of the candidate merging pairs corresponding to their association scores on each association feature dimension; Based on each association score and its corresponding weight, a weighted fusion value of all association scores is calculated, and the weighted fusion value is used as the merging score of the candidate merging pair.

[0108] Combination Figure 5 As shown, this embodiment of the present disclosure provides another document information extraction device 400. In addition to including a text box recognition module 401, an iterative execution module 402, and an extraction result determination module 403, the document information extraction device 400 also includes an image normalization module 404.

[0109] The image normalization module 404 is configured to: calculate the scaling factor required to scale the original document image proportionally to fit it completely within the preset target size frame; scale the original document image proportionally according to the scaling factor; and perform zero-padding on one side of the proportionally scaled document image until the size of the document image reaches the preset target size, thereby obtaining the target document image.

[0110] In some embodiments, the image normalization module 404 is configured to: determine the aspect ratio of the original document image; if the aspect ratio exceeds a preset aspect ratio threshold, crop the non-core area of ​​the original document image and adjust it to a preset reasonable aspect ratio range, and calculate the scaling factor required to proportionally scale the original document image after aspect ratio adjustment to fit completely within the preset target size frame; if the aspect ratio does not exceed the preset aspect ratio threshold, directly calculate the scaling factor required to proportionally scale the original document image to fit completely within the preset target size frame.

[0111] Combination Figure 6As shown, this embodiment of the disclosure provides an electronic device 600, which includes a processor 601 and a memory 602. Optionally, the electronic device 600 may further include a communication interface 603 and a bus 604. The processor 601, communication interface 603, and memory 602 can communicate with each other via the bus 604. The communication interface 603 can be used for information transmission. The processor 601 can call logical instructions in the memory 602 to execute the document information extraction method of the above embodiment.

[0112] Furthermore, the logic instructions in the aforementioned memory 602 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium.

[0113] The memory 602, as a computer-readable storage medium, can be used to store software programs and computer-executable programs, such as program instructions / modules corresponding to the methods in the embodiments of this disclosure. The processor 601 executes functional applications and data processing by running the program instructions / modules stored in the memory 602, thereby implementing the document information extraction method in the above embodiments.

[0114] The memory 602 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the terminal device. Furthermore, the memory 602 may include high-speed random access memory and may also include non-volatile memory.

[0115] This disclosure provides a storage medium storing computer program instructions, which, when executed by a processor, perform the aforementioned document information extraction method.

[0116] The technical solutions of this disclosure can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes one or more instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in this disclosure. The aforementioned storage medium can be a non-transitory storage medium, such as a USB flash drive, external hard drive, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk, etc., and other media capable of storing program code.

[0117] The foregoing description and accompanying drawings fully illustrate embodiments of this disclosure to enable those skilled in the art to practice them. Other embodiments may include structural, logical, electrical, procedural, and other changes. The embodiments represent only possible variations. Individual components and functions are optional unless explicitly required, and the order of operation may vary. Parts and features of some embodiments may be included in or replace parts and features of other embodiments. Moreover, the terminology used in this application is for describing embodiments only and is not intended to limit the claims. As used in the description of embodiments and claims, the singular forms “a,” “an,” and “the” are intended to equally include the plural forms unless the context clearly indicates otherwise. Similarly, the term “and / or” as used in this application means including one or more of the associated listed items and all possible combinations thereof. Additionally, when used in this application, the term "comprise" and its variations "comprises" and / or "comprising" refer to the presence of stated features, integrals, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof. Without further limitations, an element defined by the phrase "comprises a..." does not exclude the presence of other identical elements in the process, method, or apparatus that includes said element. In this document, each embodiment may focus on the differences from other embodiments, and similar or identical parts between embodiments can be referred to mutually. For methods, products, etc., disclosed in the embodiments, if they correspond to the method section disclosed in the embodiments, the relevant parts can be referred to the description of the method section.

[0118] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of this disclosure. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0119] The methods and products disclosed in the embodiments herein (including but not limited to devices and equipment) can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For instance, the division of units may be merely a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to implement this embodiment according to actual needs. In addition, the functional units in the embodiments of this disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0120] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than that shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. In the descriptions corresponding to the flowcharts and block diagrams in the accompanying drawings, the operations or steps corresponding to different blocks may also occur in a different order than disclosed in the description, and sometimes there is no specific order between different operations or steps. For example, two consecutive operations or steps may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. Each block in a block diagram and / or flowchart, and combinations of blocks in a block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

Claims

1. A method for extracting document information, characterized in that, include: Text recognition is performed on the target document image to obtain multiple text boxes, and these text boxes are then constructed into an initial set of text boxes according to the order of the document content. The following merge process is executed iteratively: Form a candidate merge pair by combining any two adjacent text boxes in the current set of text boxes, and obtain multiple candidate merge pairs; Calculate the association scores of the two text boxes in each candidate merge pair on each dimension of the association features; The merging score of the candidate merging pair is calculated based on the association scores of the candidate merging pairs on each association feature dimension. The candidate merge pairs with the highest merge scores that exceed the preset score threshold are selected as the target merge pairs. The two text boxes in the target merge pairs are concatenated into a new text box, and the new text box set is updated. The iteration stops when the merging score of each candidate merging pair does not exceed the score threshold, and the current set of text boxes is taken as the result of document information extraction.

2. The document information extraction method according to claim 1, characterized in that, Association features include spatial proximity. The association score between two text boxes in a candidate merge pair on the spatial proximity dimension is calculated as follows: Calculate the geometric distance and alignment deviation between the two text boxes in the candidate merge pair; The geometric distance and alignment deviation are fused, and the result of the fusion is used as the association score of the two text boxes in the candidate merge pair in the spatial proximity dimension.

3. The document information extraction method according to claim 2, characterized in that, The geometric distance between two text boxes includes at least one of the following: Vertical overlap ratio is calculated based on the intersection-union ratio of two text boxes in the vertical direction, and is used to reflect the degree of overlap between the two text boxes in the vertical direction. Horizontal spacing is calculated based on the ratio of the actual horizontal distance between the edges of two text boxes to the height of the text box characters, and is used to reflect the horizontal proximity of the two text boxes.

4. The document information extraction method according to claim 1, characterized in that, The association feature includes semantic coherence. The association score of the two text boxes in the candidate merge pair on the semantic coherence dimension is calculated as follows: Calculate the semantic relevance of the concatenated text boxes in the candidate merge pair; The semantic relevance is quantified, and the result of the quantification is used as the relevance score of the two text boxes in the candidate merging pair on the dimension of semantic coherence.

5. The document information extraction method according to claim 1, characterized in that, The association features include layout consistency. The association score of the two text boxes in the candidate merge pair on the layout consistency dimension is calculated as follows: Calculate the similarity of various layout features of two text boxes in the candidate merge pair, wherein the layout features include at least one of font size, line height, indentation and color; The similarity of various layout features is fused, and the result of the fusion is used as the association score of the two text boxes in the candidate merge pair on the layout consistency dimension.

6. The method for extracting document information according to any one of claims 1 to 5, characterized in that, Based on the association scores of the candidate merging pair across various association feature dimensions, the merging score of the candidate merging pair is calculated, including: Determine the weights of the candidate merging pairs corresponding to their association scores on each association feature dimension; Based on each association score and its corresponding weight, a weighted fusion value of all association scores is calculated, and the weighted fusion value is used as the merging score of the candidate merging pair.

7. The document information extraction method according to any one of claims 1 to 5, characterized in that, Before performing text recognition on the target document image, the following steps are also included: Calculate the scaling factor required to scale the original document image proportionally to fit it completely within the preset target size frame; The original document image is scaled proportionally according to the scaling factor; Zero-padding is performed on one side of the proportionally scaled document image until the document image reaches the preset target size, thus obtaining the target document image.

8. The document information extraction method according to claim 7, characterized in that, Calculate the scaling factor required to proportionally scale the original document image to fit completely within the preset target size frame, including: Determine the aspect ratio of the original document image; If the aspect ratio exceeds the preset aspect ratio threshold, the non-core area of ​​the original document image is cropped and adjusted to the preset reasonable aspect ratio range. The scaling factor required to scale the original document image after aspect ratio adjustment proportionally to fit completely into the preset target size frame is calculated. If the aspect ratio does not exceed the preset aspect ratio threshold, the scaling factor required to scale the original document image proportionally to fit it completely within the preset target size frame is directly calculated.

9. A document information extraction device, characterized in that, include: The text box recognition module is configured to perform text recognition on the target document image, obtain multiple text boxes, and construct an initial text box set according to the order of the document content. The iterative execution module is configured to iteratively execute the following merging process: any two adjacent text boxes in the current text box set are combined into a candidate merging pair, resulting in multiple candidate merging pairs; the association scores of the two text boxes in each candidate merging pair are calculated on each association feature dimension; based on the association scores of the candidate merging pair on each association feature dimension, the merging score of the candidate merging pair is calculated; the candidate merging pair with the highest merging score exceeding a preset score threshold is selected as the target merging pair, the two text boxes in the target merging pair are concatenated into a new text box, and the new text box set is updated. The extraction result determination module is configured to stop iterating when the merging score of each candidate merging pair does not exceed the score threshold, and to use the current set of text boxes as the document information extraction result.

10. An electronic device comprising a processor and a memory storing computer program instructions, characterized in that, The processor is configured to execute the document information extraction method as described in any one of claims 1 to 8 when running the program instructions.

11. A storage medium, characterized in that, The storage medium stores computer program instructions, which, when executed by a processor, perform the document information extraction method as described in any one of claims 1 to 8.

Images