Table text boundary adaptive information extraction method and system

Abstract

The invention provides a form text boundary adaptive information extraction method and system in the technical field of computer information processing. The method comprises the following steps: S1, identifying a title frame from a form image; s2, calculating the aspect ratio of the title frame to determine the title arrangement direction of the table titles; s3, detecting a text detection box of the table image, and performing classification prediction of a text arrangement direction on the text detection box by combining a text direction model with a title arrangement direction; s4, controlling affine transformation of the table image through the text arrangement direction and the text detection box to obtain a corrected image; s5, identifying a table title in the corrected image, and matching a table type from the table type configuration file based on the table title; s6, extracting keywords from the corrected image based on the table type; and S7, processing the keyword to extract table information. The method has the advantages that the adaptability, accuracy and efficiency of table information extraction are greatly improved.

Description

Technical Field

[0001] This invention relates to the field of computer information processing technology, and in particular to a method and system for information extraction that adapts to the boundaries of table text. Background Technology

[0002] In recent years, information extraction technology, as a key information processing method, has received widespread attention in the field of intelligent spreadsheet analysis. Its core task is to accurately identify and extract key business-value information from spreadsheets with diverse structures and layouts, covering various types such as invoices, split sheets, detail sheets, and settlement sheets. As enterprises undergo deepening digital transformation, traditional methods relying on manual spreadsheet processing are no longer sufficient to handle the demands of massive, high-concurrency data processing. Information extraction technology based on intelligent algorithms provides an efficient and scalable solution to this problem.

[0003] In practical applications, table information extraction involves not only text content recognition but also comprehensive consideration of multiple dimensions such as the overall table layout, text arrangement, and contextual semantic relationships. A typical processing flow includes: first, converting text in an image into structured text using Optical Character Recognition (OCR); then, constructing the table's coordinate matrix using table line detection algorithms; and finally, combining Natural Language Processing (NLP) and deep learning models to parse the logical relationships between fields, ultimately achieving accurate location and extraction of key fields such as invoice amount, contract signing date, and customer name. Compared to traditional rule-based key field configuration methods, this multimodal information fusion strategy significantly enhances the system's understanding of table semantics, maintaining high extraction accuracy even in complex and varied table structures.

[0004] With the rapid development of large-scale modeling technology, multimodal information extraction models, represented by Layout LM and UIE-X, have gradually become mainstream. These models achieve deep perception of table layout structures by jointly modeling text and image features. They can not only recognize text content but also combine visual information such as position, layout, color, and font to further improve the extraction accuracy and generalization ability of non-fixed-format, low-structure tables (such as VAT invoices and contracts), effectively compensating for the shortcomings of traditional methods in terms of flexibility and adaptability.

[0005] However, despite technological advancements, many systems still rely on traditional rule-based matching methods. This involves manually pre-setting keyword rules (such as amount, date, invoice number, etc.) and using regular expressions or string matching to extract fields. While this method remains feasible for formatted tables, it has significant drawbacks: rule formulation is time-consuming and labor-intensive, it heavily relies on prior domain knowledge, it has weak ability to distinguish similar fields, and post-processing optimization is difficult. More importantly, traditional methods often ignore the spatial location and layout structure of text, easily leading to mismatches or missed matches, resulting in low reliability of the extracted results.

[0006] Furthermore, despite the superior accuracy of large models, they still face numerous challenges in large-scale deployment. For example, in business scenarios with high concurrency or real-time requirements, model inference is slow, computational resources are consumed excessively, and overall call efficiency is low. Simultaneously, the poor interpretability of the model's decision-making process also limits its widespread application in industrial environments. Therefore, while fully leveraging the advantages of high-precision extraction from large models, optimizing their inference performance and improving processing efficiency and interpretability have become critical issues that urgently need to be addressed in this field.

[0007] In summary, how to provide an information extraction method and system that adapts to the boundaries of table text to improve the adaptability, accuracy, and efficiency of table information extraction has become an urgent technical problem to be solved. Summary of the Invention

[0008] The technical problem to be solved by this invention is to provide a method and system for information extraction that adapts to the boundaries of table text, thereby improving the adaptability, accuracy and efficiency of table information extraction.

[0009] In a first aspect, the present invention provides an information extraction method with adaptive table text boundaries, comprising the following steps:

[0010] Step S1: Obtain the table image from which information is to be extracted, identify the table title region from the table image using an instance segmentation model, and then locate the title box surrounding the table title region;

[0011] Step S2: Calculate the aspect ratio of the title box to determine the title arrangement direction of the table title;

[0012] Step S3: Detect all text detection boxes in the table image using a text detection model, and perform text arrangement direction prediction for all text detection boxes using a text direction model combined with the title arrangement direction;

[0013] Step S4: Correct the table image using the text arrangement direction and all text detection boxes to obtain a corrected image;

[0014] Step S5: Identify the table title within the title box in the corrected image using a text recognition model, and match the table type from a predefined table type configuration file based on the table title;

[0015] Step S6: Extract keywords from the corrected image based on the table type;

[0016] Step S7: Process the extracted keywords using a post-processing function to extract table information.

[0017] Furthermore, in step S1, the instance segmentation model adopts either the YOLO model or the Mask R-CNN model;

[0018] In step S2, the title arrangement direction is either horizontal or vertical.

[0019] In step S3, the text detection model adopts either the DB text detection model or the DB++ text detection model.

[0020] Furthermore, step S4 specifically includes:

[0021] By controlling the text arrangement direction and the average centerline slope of all text detection boxes, the table image is subjected to an affine transformation around the image center point to obtain a corrected image.

[0022] Furthermore, in step S5, the text recognition model adopts either the SVTR model or the C-RNN model;

[0023] Each of the aforementioned table types corresponds to a combination of table elements; the table elements include at least a table title, table lines, text content, seal, fingerprint, and signature.

[0024] Furthermore, step S6 specifically includes:

[0025] Based on the table type, keywords are extracted from the corrected image by combining direct extraction algorithm and relation extraction algorithm;

[0026] In step S7, the post-processing function includes at least a field mapping function, a text content replacement function, a symbol cleanup function, and a normalization function.

[0027] Secondly, the present invention provides an information extraction system with adaptive table text boundaries, comprising the following modules:

[0028] The title box localization module is used to acquire a table image from which information is to be extracted, identify the table title region from the table image using an instance segmentation model, and then locate the title box surrounding the table title region.

[0029] The title arrangement direction recognition module is used to calculate the aspect ratio of the title box to determine the title arrangement direction of the table title;

[0030] The text arrangement direction recognition module is used to detect all text detection boxes in the table image through a text detection model, and to classify and predict the text arrangement direction of all text detection boxes by combining the text direction model with the title arrangement direction.

[0031] The table image correction module is used to correct the table image using the text arrangement direction and all text detection boxes to obtain a corrected image;

[0032] The table type matching module is used to identify the table title within the title box in the corrected image through a text recognition model, and match the table type from a predefined table type configuration file based on the table title;

[0033] The keyword extraction module is used to extract keywords from the corrected image based on the table type;

[0034] The table information extraction module is used to process the extracted keywords through a post-processing function to extract table information.

[0035] Furthermore, in the title box localization module, the instance segmentation model adopts either the YOLO model or the Mask R-CNN model;

[0036] In the title arrangement direction recognition module, the title arrangement direction is either horizontal or vertical.

[0037] In the text alignment direction recognition module, the text detection model adopts either the DB text detection model or the DB++ text detection model.

[0038] Furthermore, the table image correction module is specifically used for:

[0039] By controlling the text arrangement direction and the average centerline slope of all text detection boxes, the table image is subjected to an affine transformation around the image center point to obtain a corrected image.

[0040] Furthermore, in the table type matching module, the text recognition model adopts either the SVTR model or the C-RNN model;

[0041] Each of the aforementioned table types corresponds to a combination of table elements; the table elements include at least a table title, table lines, text content, seal, fingerprint, and signature.

[0042] Furthermore, the keyword extraction module is specifically used for:

[0043] Based on the table type, keywords are extracted from the corrected image by combining direct extraction algorithm and relation extraction algorithm;

[0044] In the table information extraction module, the post-processing function includes at least a field mapping function, a text content replacement function, a symbol cleanup function, and a standardization function.

[0045] The advantages of this invention are:

[0046] 1. By acquiring the table image from which information is to be extracted, an instance segmentation model is used to identify the table title region from the table image, thereby locating the title box surrounding the table title region. Then, the aspect ratio of the title box is calculated to determine the title arrangement direction. A text detection model is used to detect all text detection boxes in the table image, and a text direction model, combined with the title arrangement direction, is used to classify and predict the text arrangement direction of all text detection boxes. Next, the table image is corrected using the text arrangement direction and text detection boxes to obtain a corrected image. A text recognition model is used to identify the table title within the title box in the corrected image. Based on the table title, the table type is matched from a predefined table type configuration file. Keywords are extracted from the corrected image based on the table type. Finally, a post-processing function is used to process the extracted keywords to extract table information. In other words, firstly, instance segmentation is used... The segmentation model locates the title box and dynamically determines the title arrangement direction (horizontal / vertical) based on the aspect ratio. It combines the orientation classification prediction of the text detection box and affine transformation to achieve adaptive correction of the table image, solving the mismatch problem caused by the neglect of layout structure in traditional methods. Secondly, it matches the corrected title content through a predefined table type configuration file, integrates direct extraction and relation extraction algorithms to accurately locate keywords, and introduces post-processing functions such as field mapping and symbol cleaning to automatically optimize the output, avoiding reliance on manual rules. Lightweight models (such as YOLO and DB++) are used to process the detection, orientation classification and recognition tasks in stages, replacing the high computational load of end-to-end large models. While ensuring the accuracy of spatial semantic understanding, it significantly reduces resource consumption, efficiently adapts to diverse table structures and high-concurrency scenarios, and ultimately greatly improves the adaptability, accuracy and efficiency of table information extraction.

[0047] 2. This invention cleverly integrates multiple models and coordinate positioning, combining various features such as position coordinates and text content in the table. This ensures that the interpretability and computational speed of the extraction algorithm are maintained without sacrificing keyword extraction accuracy. Furthermore, it supports a flexible configuration mechanism based on templates (table type configuration files). Users can customize the keyword search area and matching rules according to the structural characteristics of different tables and business requirements, effectively avoiding mismatches caused by global searches and improving robustness and adaptability. This not only solves the problem of insufficient generalization ability of existing methods when facing diverse table formats, but also provides solid technical support for achieving rapid deployment, dynamic adjustment, and intelligent maintenance.

[0048] 3. Compared to traditional rule-matching methods and general visual multimodal large-scale model information extraction methods, this invention significantly improves the efficiency of information extraction while accurately extracting keyword information from tables. By introducing a table text keyword position search interval configuration mechanism based on human priors, it effectively reduces the dependence on large-scale labeled data and significantly reduces the consumption of computational resources during the inference process, thereby accelerating processing speed and making the information extraction process more efficient and concise. The configurable design significantly enhances the interpretability of model behavior, making it easier for developers and users to understand its operating logic, which is of great significance for system debugging, error troubleshooting, and performance optimization. Furthermore, compared to general visual multimodal large-scale models, it significantly reduces the demand for GPU memory resources, enabling efficient deployment and operation in resource-constrained production environments, significantly improving the space utilization of computing resources and the overall scalability of the system. It also effectively solves several bottlenecks in existing technologies in terms of data dependence, computational overhead, interpretability, and resource utilization, demonstrating significant advantages in accuracy, efficiency, flexibility, and engineering practicality.

[0049] 4. Compared to the extraction efficiency of traditional rules and general visual multimodal large models for matching key fields, this method can accurately locate the extracted keywords by combining table layout information while ensuring efficient inference extraction speed.

[0050] 5. By setting pre-defined table keyword layout information into the extraction configuration file, table images can be quickly configured for different scenarios according to requirements. Appropriate extraction modes such as "direct extraction" and "relational extraction" can be selected for keywords of different table images. Based on the extraction results, personalized processing methods can be used to adapt to the extraction of different keywords.

[0051] 6. By combining the design of keyword rules across three dimensions, the coordinates of text detection boxes, and the recognition of text detection boxes, a search boundary covering the extracted keywords is generated. Based on this, the task of extracting relational keywords is implemented. By decomposing the main key into multiple subkeys, their table text extraction boundaries are found. A text box slope matching algorithm is designed to find text elements in the same row or column, thereby realizing the extraction of relational tasks. In complex scenarios, this algorithm can select the appropriate extraction method according to the extraction needs of different key fields to achieve efficient extraction results. Attached Figure Description

[0052] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0053] Figure 1 This is a flowchart of an information extraction method for adaptive table text boundaries according to the present invention.

[0054] Figure 2 This is a schematic diagram of the structure of an information extraction system for adaptive table text boundaries according to the present invention.

[0055] Figure 3 This is one of the schematic diagrams for extracting table information according to the present invention.

[0056] Figure 4 This is the second schematic diagram of the table information extraction of the present invention.

[0057] Figure 5 This is the third schematic diagram of the table information extraction of the present invention. Detailed Implementation

[0058] The overall approach of the technical solution in this application is as follows: An instance segmentation model is used to locate the title box and dynamically determine the title arrangement direction based on the aspect ratio. This is combined with orientation classification prediction of the text detection box and affine transformation to achieve adaptive correction of the table image, solving the mismatch problem caused by ignoring the layout structure in traditional methods. Secondly, the corrected title content is matched using a predefined table type configuration file. Direct extraction and relation extraction algorithms are integrated to accurately locate keywords, and post-processing functions such as field mapping and symbol cleanup are introduced to automatically optimize the output, avoiding reliance on manual rules. A lightweight model is used to process the detection, orientation classification, and recognition tasks in stages, replacing the high computational load of the end-to-end large model. This significantly reduces resource consumption while ensuring the accuracy of spatial semantic understanding, efficiently adapting to diverse table structures and high-concurrency scenarios, thereby improving the adaptability, accuracy, and efficiency of table information extraction.

[0059] Please refer to Figures 1 to 5 As shown, a preferred embodiment of the information extraction method for adaptive table text boundaries of the present invention includes the following steps:

[0060] Step S1: Obtain the table image from which information is to be extracted, identify the table title region from the table image using an instance segmentation model, and then locate the title box surrounding the table title region;

[0061] Step S2: Calculate the aspect ratio of the title box to determine the title arrangement direction of the table title;

[0062] Step S3: Detect all text detection boxes in the table image using a text detection model, and perform text arrangement direction prediction for all text detection boxes using a text direction model combined with the title arrangement direction;

[0063] Step S4: Correct the table image by using the text arrangement direction and all text detection boxes to obtain a corrected image; that is, adjust the input table image to the positive text direction to ensure that the subsequent model can accurately read the content in the detection boxes;

[0064] Step S5: Identify the table title within the title box in the corrected image using a text recognition model, and match the table type from a predefined table type configuration file based on the table title;

[0065] Step S6: Extract keywords from the corrected image based on the table type;

[0066] Step S7: Process the extracted keywords using a post-processing function to extract table information. In step S1, the instance segmentation model uses either the YOLO model or the Mask R-CNN model.

[0067] In step S2, the title arrangement direction is either horizontal or vertical.

[0068] In step S3, the text detection model adopts either the DB text detection model or the DB++ text detection model.

[0069] Step S4 specifically involves:

[0070] By controlling the text arrangement direction and the average centerline slope of all text detection boxes, the table image is subjected to an affine transformation around the image center point to obtain a corrected image.

[0071] There are two methods for correcting table images: one is to directly use an end-to-end image correction model; the other is to combine a text direction prediction model with image post-processing techniques.

[0072] End-to-end image correction models typically employ a large encoder-decoder architecture to complete the image correction task. While this approach performs well in extreme cases, its complex structure leads to lengthy inference processes. Considering that in real-world applications the algorithm may be called frequently and the number of images uploaded by users requiring correction may be limited, this high-cost correction method may result in wasted resources.

[0073] In contrast, combining text direction prediction with image processing techniques is more efficient, especially for table images. This is based on the premise that when the text is read from left to right, the horizontal lines of the text boxes within the table are roughly parallel to the table's horizontal line. First, an instance segmentation model is used to identify the table title area, thereby calculating the aspect ratio of the title box to determine the title's orientation. Next, a text direction model is applied to classify and predict all text detection boxes, where "0" indicates the text may be left- or right-handed, and "1" represents right- or right-handed. Based on the title's orientation, the overall orientation of the table can be determined.

[0074] However, simply converting the image to the forward orientation is insufficient to guarantee that the image is completely free of tilt. Therefore, by analyzing the average slope of the center line of all text boxes within the table and performing an affine transformation around the image center point based on this slope, the final corrected image is obtained. This method not only meets the needs of most practical applications but also offers faster processing speeds compared to end-to-end correction algorithms.

[0075] In step S5, the text recognition model adopts either the SVTR model or the C-RNN model;

[0076] Each of the aforementioned table types corresponds to a combination of table elements; the table elements include at least a table title, table lines, text content, seal, fingerprint, and signature.

[0077] Since table images are typically composed of multiple table elements, the combination of these elements forms a table image that carries various types of information. To better categorize the functional attributes of these table images, the most direct and effective way is to use table titles to distinguish different table template types. This method allows for quick identification and classification of the business scenarios or uses corresponding to the tables.

[0078] In practical applications, the table title region is detected by a rotational object detection model, and the text recognition models such as SVTR and C-RNN are used to recognize the text in the table title region. Then, a predefined table type configuration file is used for matching and judgment. The table type configuration file contains various key fields that need to be extracted, and each table type corresponds to a specific regular expression text.

[0079] By comparing the identified title text with regular expressions in the table type configuration file, the template type of the table can be quickly determined. This mechanism not only efficiently distinguishes different types of table images but also effectively filters out meaningless or distracting image content, thereby improving the accuracy and efficiency of the overall processing workflow. For the large number of table images uploaded by users, this method helps to achieve standardized management and intelligent review, improving the overall quality and response speed of information extraction.

[0080] Step S6 specifically involves:

[0081] Based on the table type, keywords are extracted from the corrected image by combining a direct extraction algorithm and a relation extraction algorithm (table relation extraction function);

[0082] The Table Text Boundary Extraction Module (TTBEM) comprehensively utilizes the coordinate space information of text boxes, the overall layout structure of the table, and the text content itself, combined with flexible manual configuration, to achieve efficient and accurate extraction of key fields from tables. The TTBEM module includes two main functions: "direct extraction" and "relational extraction," as shown in the formula:

[0083]

[0084] in, Indicates keywords; This indicates a direct extraction algorithm; Represents the relation extraction algorithm; P k This represents the k-th table text search box; k represents the number of sub-keywords used to extract the keyword, and each sub-keyword will generate a different table text search box.

[0085] "Direct extraction" means that only one key subkey is extracted, so k equals 1. Extraction yields the keyword extraction results, and this extracted text contains only one element. "Relational extraction" means extracting a keyword with at least one sub-keyword. It generates k table text search boxes, which typically need to extract multiple fields, based on the characteristics of the table relational extraction task. Output relational extraction results based on the order of table attributes (from left to right or from top to bottom) and the extracted text in the k table text search boxes.

[0086] The core of TTBEM lies in the table text search box, which sets a positive table image χ. img When performing information extraction tasks, the detection results of the text detection model are required. The method for obtaining these results is as follows:

[0087]

[0088] Where, d i The text detection model represents the text from the table image χ. img The i-th detected text detection box; i represents the index of the text detection box; This represents a text detection model. This represents the set of text detection boxes. The text recognition model then processes all text detection boxes... Perform individual identification:

[0089]

[0090] in, Represents a text recognition model; r i The text detection box d represents i Entering the text recognition model The text recognition results; This represents the set of text recognition. Based on the above information, TTBEM uses its configuration file and internal algorithm to cleverly design the text extraction boundary P for this keyword. P is composed of the specific coordinate values ​​of four points: the x and y coordinates of the upper left anchor point and the x and y coordinates of the lower right anchor point. The formula for obtaining P is:

[0091] p j =ψ(r) j ,t j ,p j ,φ j ,χ det ,χ rec )and p j ∈P;

[0092] Where, p j It is a specific x or y coordinate value among the four points mentioned above, and p j Belongs to P; ψ represents the decimation function of TTBEM; parameter r j The text rule representing the anchor point of the j-th table text search box; t j This indicates that retrieving the text at the anchor point of the j-th table text search box requires using the vertex coordinates of its corresponding text detection box; p jφ represents the expansion or reduction ratio of the anchor point coordinates of the j-th table text search box; j An auxiliary function representing the anchor point of the j-th table text search box;

[0093] Based on the above formula, the approximate text search range for the keyword can be determined. Using the "direct extraction" algorithm, the value corresponding to the keyword can be found within this range. In practical scenarios, setting only a text extraction boundary P is insufficient. The generation of the text extraction boundary depends on the text rule matching of its internal anchor point coordinates. If the text rule of the anchor point cannot match χ... rec If the text matching is incorrect, then no text extraction boundary P will be generated. To address this, multiple text extraction boundaries P will be used for a key field in the configuration. The most direct difference between these text extraction boundaries P lies in the text rules of their internal anchor points. In complex scenarios, a single anchor point text rule may result in multiple texts being matched, which greatly hinders the accuracy of table information extraction. For example, if the required anchor point text rule is "^Total$", but multiple "Total" fields appear in the table image, then the final anchor point will fall on the first matched "Total" field. This is because the extraction algorithm only uses absolute layout information when calculating anchor point coordinates and does not incorporate relative layout information. The auxiliary function φ... i By incorporating relative layout information for target matching anchor points, the optimal anchor point matching field can be obtained simply by manually selecting a unique field in the table image and calculating the "distance relationship" between these two fields, thus eliminating the problem of field matching confusion. To ensure the timeliness of calculating multiple text extraction boundaries, anchor points using the same rules are memorized to avoid redundant calculations. When encountering the same text rule again, the text detection box and its corresponding text can be quickly obtained, ensuring the high efficiency of the entire extraction algorithm.

[0094] Another function of TTBEM is "relation extraction." For relational extraction fields, it uses multiple subkeys, which means it generates k P values, and then uses the TTBEM module... The P values ​​for these different sub-keywords are arranged according to the order of the table attributes. Assuming the table attributes are arranged from left to right, the relational fields to be extracted include four sub-keyword fields: "Minimum", "End Code", "Electricity (kWh)", and "Meter Number". These sub-keyword fields are in the same row as the "Settlement Amount (Yuan)" field. Calculating the boundaries of these four table texts will generate four P values. Since the final returned relational format is a combination of elements in the same row, it will be [["44899.26", "45482.25", "3752.00", "A51223664580"], ["125698.36", "365.26", "1562.1", "A51223664581"]...]. It will compare the slope T of the table image slope Match elements in the same row with the center line of the text detection box within each sub-keyword table text search box.

[0095] In step S7, the post-processing function includes at least a field mapping function, a text content replacement function, a symbol cleanup function, and a normalization function.

[0096] In information extraction tasks, the keywords (values) extracted by the model often cannot be used directly. In many cases, these raw extraction results require a series of targeted post-processing operations to be transformed into semantically accurate, business-usable standard text output. To solve this problem, it is necessary to perform fine-grained optimization and standardization of the extraction results through flexible configuration.

[0097] This involves combining multiple manually defined post-processing functions and configuration files to dynamically specify which post-processing logic should be applied to corresponding fields after extraction, based on the structural characteristics of different table templates. These post-processing functions can be combined or extended according to actual needs to meet diverse business scenarios. Specifically, the configuration file defines the fields to be extracted for each table template and binds one or more applicable post-processing functions to each field. After keyword extraction is complete, the corresponding post-processing function is automatically called according to the current table type to perform operations such as correcting the original keywords, format conversion, keyword replacement, and unit standardization, ultimately outputting high-quality, structurally consistent standard text.

[0098] This design not only effectively improves the accuracy and usability of the extraction results, but also offers excellent flexibility and scalability. For different types of tabular images, simply updating the configuration file and the corresponding post-processing functions allows for rapid adaptation to new template structures and business requirements, greatly enhancing the system's adaptability and practicality.

[0099] A preferred embodiment of the present invention provides an information extraction system for adaptive table text boundaries, comprising the following modules:

[0100] The title box localization module is used to acquire a table image from which information is to be extracted, identify the table title region from the table image using an instance segmentation model, and then locate the title box surrounding the table title region.

[0101] The title arrangement direction recognition module is used to calculate the aspect ratio of the title box to determine the title arrangement direction of the table title;

[0102] The text arrangement direction recognition module is used to detect all text detection boxes in the table image through a text detection model, and to classify and predict the text arrangement direction of all text detection boxes by combining the text direction model with the title arrangement direction.

[0103] The table image correction module is used to correct the table image by using the text arrangement direction and all text detection boxes to obtain a corrected image; that is, to adjust the input table image to the positive text direction to ensure that the subsequent model can accurately read the content in the detection boxes;

[0104] The table type matching module is used to identify the table title within the title box in the corrected image through a text recognition model, and match the table type from a predefined table type configuration file based on the table title;

[0105] The keyword extraction module is used to extract keywords from the corrected image based on the table type;

[0106] The table information extraction module is used to process the extracted keywords through a post-processing function to extract table information.

[0107] In the title box localization module, the instance segmentation model adopts either the YOLO model or the Mask R-CNN model;

[0108] In the title arrangement direction recognition module, the title arrangement direction is either horizontal or vertical.

[0109] In the text alignment direction recognition module, the text detection model adopts either the DB text detection model or the DB++ text detection model.

[0110] The table image correction module is specifically used for:

[0111] By controlling the text arrangement direction and the average centerline slope of all text detection boxes, the table image is subjected to an affine transformation around the image center point to obtain a corrected image.

[0112] There are two methods for correcting table images: one is to directly use an end-to-end image correction model; the other is to combine a text direction prediction model with image post-processing techniques.

[0113] End-to-end image correction models typically employ a large encoder-decoder architecture to complete the image correction task. While this approach performs well in extreme cases, its complex structure leads to lengthy inference processes. Considering that in real-world applications the algorithm may be called frequently and the number of images uploaded by users requiring correction may be limited, this high-cost correction method may result in wasted resources.

[0114] In contrast, combining text direction prediction with image processing techniques is more efficient, especially for table images. This is based on the premise that when the text is read from left to right, the horizontal lines of the text boxes within the table are roughly parallel to the table's horizontal line. First, an instance segmentation model is used to identify the table title area, thereby calculating the aspect ratio of the title box to determine the title's orientation. Next, a text direction model is applied to classify and predict all text detection boxes, where "0" indicates the text may be left- or right-handed, and "1" represents right- or right-handed. Based on the title's orientation, the overall orientation of the table can be determined.

[0115] However, simply converting the image to the forward orientation is insufficient to guarantee that the image is completely free of tilt. Therefore, by analyzing the average slope of the center line of all text boxes within the table and performing an affine transformation around the image center point based on this slope, the final corrected image is obtained. This method not only meets the needs of most practical applications but also offers faster processing speeds compared to end-to-end correction algorithms.

[0116] In the table type matching module, the text recognition model adopts either the SVTR model or the C-RNN model;

[0117] Each of the aforementioned table types corresponds to a combination of table elements; the table elements include at least a table title, table lines, text content, seal, fingerprint, and signature.

[0118] Since table images are typically composed of multiple table elements, the combination of these elements forms a table image that carries various types of information. To better categorize the functional attributes of these table images, the most direct and effective way is to use table titles to distinguish different table template types. This method allows for quick identification and classification of the business scenarios or uses corresponding to the tables.

[0119] In practical applications, the table title region is detected by a rotational object detection model, and the text recognition models such as SVTR and C-RNN are used to recognize the text in the table title region. Then, a predefined table type configuration file is used for matching and judgment. The table type configuration file contains various key fields that need to be extracted, and each table type corresponds to a specific regular expression text.

[0120] By comparing the identified title text with regular expressions in the table type configuration file, the template type of the table can be quickly determined. This mechanism not only efficiently distinguishes different types of table images but also effectively filters out meaningless or distracting image content, thereby improving the accuracy and efficiency of the overall processing workflow. For the large number of table images uploaded by users, this method helps to achieve standardized management and intelligent review, improving the overall quality and response speed of information extraction.

[0121] The keyword extraction module is specifically used for:

[0122] Based on the table type, keywords are extracted from the corrected image by combining a direct extraction algorithm and a relation extraction algorithm (table relation extraction function);

[0123] The Table Text Boundary Extraction Module (TTBEM) comprehensively utilizes the coordinate space information of text boxes, the overall layout structure of the table, and the text content itself, combined with flexible manual configuration, to achieve efficient and accurate extraction of key fields from tables. The TTBEM module includes two main functions: "direct extraction" and "relational extraction," as shown in the formula:

[0124]

[0125] in, Indicates keywords; This indicates a direct extraction algorithm; Represents the relation extraction algorithm; P k This represents the k-th table text search box; k represents the number of sub-keywords used to extract the keyword, and each sub-keyword will generate a different table text search box.

[0126] "Direct extraction" means that only one key subkey is extracted, so k equals 1. Extraction yields the keyword extraction results, and this extracted text contains only one element. "Relational extraction" means extracting a keyword with at least one sub-keyword. It generates k table text search boxes, which typically need to extract multiple fields, based on the characteristics of the table relational extraction task. Output relational extraction results based on the order of table attributes (from left to right or from top to bottom) and the extracted text in the k table text search boxes.

[0127] The core of TTBEM lies in the table text search box, which sets a positive table image χ. ima When performing information extraction tasks, the detection results of the text detection model are required. The method for obtaining these results is as follows:

[0128]

[0129] Where, d i The text detection model represents the text from the table image χ. img The i-th detected text detection box; i represents the index of the text detection box; This represents a text detection model. This represents the set of text detection boxes. The text recognition model then processes all text detection boxes... Perform individual identification:

[0130]

[0131] in, Represents a text recognition model; r i The text detection box d represents i Entering the text recognition model The text recognition results; This represents the set of text recognition. Based on the above information, TTBEM uses its configuration file and internal algorithm to cleverly design the text extraction boundary P for this keyword. P is composed of the specific coordinate values ​​of four points: the x and y coordinates of the upper left anchor point and the x and y coordinates of the lower right anchor point. The formula for obtaining P is:

[0132] p j =ψ(r) j ,t j ,p j ,φ j ,χ det ,χ rec )and p j ∈P;

[0133] Where, p j It is a specific x or y coordinate value among the four points mentioned above, and p j Belongs to P; ψ represents the decimation function of TTBEM; parameter r j The text rule representing the anchor point of the j-th table text search box; t j This indicates that retrieving the text at the anchor point of the j-th table text search box requires using the vertex coordinates of its corresponding text detection box; p jφ represents the expansion or reduction ratio of the anchor point coordinates of the j-th table text search box; j An auxiliary function representing the anchor point of the j-th table text search box;

[0134] Based on the above formula, the approximate text search range for the keyword can be determined. Using the "direct extraction" algorithm, the value corresponding to the keyword can be found within this range. In practical scenarios, setting only a text extraction boundary P is insufficient. The generation of the text extraction boundary depends on the text rule matching of its internal anchor point coordinates. If the text rule of the anchor point cannot match χ... rec If the text matching is incorrect, then no text extraction boundary P will be generated. To address this, multiple text extraction boundaries P will be used for a key field in the configuration. The most direct difference between these text extraction boundaries P lies in the text rules of their internal anchor points. In complex scenarios, a single anchor point text rule may result in multiple texts being matched, which greatly hinders the accuracy of table information extraction. For example, if the required anchor point text rule is "^Total$", but multiple "Total" fields appear in the table image, then the final anchor point will fall on the first matched "Total" field. This is because the extraction algorithm only uses absolute layout information when calculating anchor point coordinates and does not incorporate relative layout information. The auxiliary function φ... j By incorporating relative layout information for target matching anchor points, the optimal anchor point matching field can be obtained simply by manually selecting a unique field in the table image and calculating the "distance relationship" between these two fields, thus eliminating the problem of field matching confusion. To ensure the timeliness of calculating multiple text extraction boundaries, anchor points using the same rules are memorized to avoid redundant calculations. When encountering the same text rule again, the text detection box and its corresponding text can be quickly obtained, ensuring the high efficiency of the entire extraction algorithm.

[0135] Another function of TTBEM is "relation extraction." For relational extraction fields, it uses multiple subkeys, which means it generates k P values, and then uses the TTBEM module... The P values ​​for these different sub-keywords are arranged according to the order of the table attributes. Assuming the table attributes are arranged from left to right, the relational fields to be extracted include four sub-keyword fields: "Minimum", "End Code", "Electricity (kWh)", and "Meter Number". These sub-keyword fields are in the same row as the "Settlement Amount (Yuan)" field. Calculating the boundaries of these four table texts will generate four P values. Since the final returned relational format is a combination of elements in the same row, it will be [["44899.26", "45482.25", "3752.00", "A51223664580"], ["125698.36", "365.26", "1562.1", "A51223664581"]...]. It will compare the slope T of the table image slope Match elements in the same row with the center line of the text detection box within each sub-keyword table text search box.

[0136] In the table information extraction module, the post-processing function includes at least a field mapping function, a text content replacement function, a symbol cleanup function, and a standardization function.

[0137] In information extraction tasks, the keywords (values) extracted by the model often cannot be used directly. In many cases, these raw extraction results require a series of targeted post-processing operations to be transformed into semantically accurate, business-usable standard text output. To solve this problem, it is necessary to perform fine-grained optimization and standardization of the extraction results through flexible configuration.

[0138] This involves combining multiple manually defined post-processing functions and configuration files to dynamically specify which post-processing logic should be applied to corresponding fields after extraction, based on the structural characteristics of different table templates. These post-processing functions can be combined or extended according to actual needs to meet diverse business scenarios. Specifically, the configuration file defines the fields to be extracted for each table template and binds one or more applicable post-processing functions to each field. After keyword extraction is complete, the corresponding post-processing function is automatically called according to the current table type to perform operations such as correcting the original keywords, format conversion, keyword replacement, and unit standardization, ultimately outputting high-quality, structurally consistent standard text.

[0139] This design not only effectively improves the accuracy and usability of the extraction results, but also offers excellent flexibility and scalability. For different types of tabular images, simply updating the configuration file and the corresponding post-processing functions allows for rapid adaptation to new template structures and business requirements, greatly enhancing the system's adaptability and practicality.

[0140] In summary, the advantages of this invention are as follows:

[0141] 1. By acquiring the table image from which information is to be extracted, an instance segmentation model is used to identify the table title region from the table image, thereby locating the title box surrounding the table title region. Then, the aspect ratio of the title box is calculated to determine the title arrangement direction. A text detection model is used to detect all text detection boxes in the table image, and a text direction model, combined with the title arrangement direction, is used to classify and predict the text arrangement direction of all text detection boxes. Next, the table image is corrected using the text arrangement direction and text detection boxes to obtain a corrected image. A text recognition model is used to identify the table title within the title box in the corrected image. Based on the table title, the table type is matched from a predefined table type configuration file. Keywords are extracted from the corrected image based on the table type. Finally, a post-processing function is used to process the extracted keywords to extract table information. In other words, firstly, instance segmentation is used... The segmentation model locates the title box and dynamically determines the title arrangement direction (horizontal / vertical) based on the aspect ratio. It combines the orientation classification prediction of the text detection box and affine transformation to achieve adaptive correction of the table image, solving the mismatch problem caused by the neglect of layout structure in traditional methods. Secondly, it matches the corrected title content through a predefined table type configuration file, integrates direct extraction and relation extraction algorithms to accurately locate keywords, and introduces post-processing functions such as field mapping and symbol cleaning to automatically optimize the output, avoiding reliance on manual rules. Lightweight models (such as YOLO and DB++) are used to process the detection, orientation classification and recognition tasks in stages, replacing the high computational load of end-to-end large models. While ensuring the accuracy of spatial semantic understanding, it significantly reduces resource consumption, efficiently adapts to diverse table structures and high-concurrency scenarios, and ultimately greatly improves the adaptability, accuracy and efficiency of table information extraction.

[0142] 2. This invention cleverly integrates multiple models and coordinate positioning, combining various features such as position coordinates and text content in the table. This ensures that the interpretability and computational speed of the extraction algorithm are maintained without sacrificing keyword extraction accuracy. Furthermore, it supports a flexible configuration mechanism based on templates (table type configuration files). Users can customize the keyword search area and matching rules according to the structural characteristics of different tables and business requirements, effectively avoiding mismatches caused by global searches and improving robustness and adaptability. This not only solves the problem of insufficient generalization ability of existing methods when facing diverse table formats, but also provides solid technical support for achieving rapid deployment, dynamic adjustment, and intelligent maintenance.

[0143] 3. Compared to traditional rule-matching methods and general visual multimodal large-scale model information extraction methods, this invention significantly improves the efficiency of information extraction while accurately extracting keyword information from tables. By introducing a table text keyword position search interval configuration mechanism based on human priors, it effectively reduces the dependence on large-scale labeled data and significantly reduces the consumption of computational resources during the inference process, thereby accelerating processing speed and making the information extraction process more efficient and concise. The configurable design significantly enhances the interpretability of model behavior, making it easier for developers and users to understand its operating logic, which is of great significance for system debugging, error troubleshooting, and performance optimization. Furthermore, compared to general visual multimodal large-scale models, it significantly reduces the demand for GPU memory resources, enabling efficient deployment and operation in resource-constrained production environments, significantly improving the space utilization of computing resources and the overall scalability of the system. It also effectively solves several bottlenecks in existing technologies in terms of data dependence, computational overhead, interpretability, and resource utilization, demonstrating significant advantages in accuracy, efficiency, flexibility, and engineering practicality.

[0144] 4. Compared to the extraction efficiency of traditional rules and general visual multimodal large models for matching key fields, this method can accurately locate the extracted keywords by combining table layout information while ensuring efficient inference extraction speed.

[0145] 5. By setting pre-defined table keyword layout information into the extraction configuration file, table images can be quickly configured for different scenarios according to requirements. Appropriate extraction modes such as "direct extraction" and "relational extraction" can be selected for keywords of different table images. Based on the extraction results, personalized processing methods can be used to adapt to the extraction of different keywords.

[0146] 6. By combining the design of keyword rules across three dimensions, the coordinates of text detection boxes, and the recognition of text detection boxes, a search boundary covering the extracted keywords is generated. Based on this, the task of extracting relational keywords is implemented. By decomposing the main key into multiple subkeys, their table text extraction boundaries are found. A text box slope matching algorithm is designed to find text elements in the same row or column, thereby realizing the extraction of relational tasks. In complex scenarios, this algorithm can select the appropriate extraction method according to the extraction needs of different key fields to achieve efficient extraction results.

[0147] While specific embodiments of the present invention have been described above, those skilled in the art should understand that the specific embodiments described are merely illustrative and not intended to limit the scope of the present invention. Equivalent modifications and variations made by those skilled in the art in accordance with the spirit of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A method for information extraction that adapts to table text boundaries, characterized in that: Includes the following steps: Step S1: Obtain the table image from which information is to be extracted, identify the table title region from the table image using an instance segmentation model, and then locate the title box surrounding the table title region; Step S2: Calculate the aspect ratio of the title box to determine the title arrangement direction of the table title; Step S3: Detect all text detection boxes in the table image using a text detection model, and perform text arrangement direction prediction for all text detection boxes using a text direction model combined with the title arrangement direction; Step S4: Correct the table image using the text arrangement direction and all text detection boxes to obtain a corrected image; Step S5: Identify the table title within the title box in the corrected image using a text recognition model, and match the table type from a predefined table type configuration file based on the table title; Step S6: Extract keywords from the corrected image based on the table type; Step S7: Process the extracted keywords using a post-processing function to extract table information.

2. The information extraction method with adaptive table text boundaries as described in claim 1, characterized in that: In step S1, the instance segmentation model adopts either the YOLO model or the Mask R-CNN model; In step S2, the title arrangement direction is either horizontal or vertical. In step S3, the text detection model adopts either the DB text detection model or the DB++ text detection model.

3. The information extraction method with adaptive table text boundaries as described in claim 1, characterized in that: Step S4 specifically involves: By controlling the text arrangement direction and the average centerline slope of all text detection boxes, the table image is subjected to an affine transformation around the image center point to obtain a corrected image.

4. The information extraction method with adaptive table text boundaries as described in claim 1, characterized in that: In step S5, the text recognition model adopts either the SVTR model or the C-RNN model; Each of the aforementioned table types corresponds to a combination of table elements; the table elements include at least a table title, table lines, text content, seal, fingerprint, and signature.

5. The information extraction method with adaptive table text boundaries as described in claim 1, characterized in that: Step S6 specifically involves: Based on the table type, keywords are extracted from the corrected image by combining direct extraction algorithm and relation extraction algorithm; In step S7, the post-processing function includes at least a field mapping function, a text content replacement function, a symbol cleanup function, and a normalization function.

6. An information extraction system with adaptive table text boundaries, characterized in that: Includes the following modules: The title box localization module is used to acquire a table image from which information is to be extracted, identify the table title region from the table image using an instance segmentation model, and then locate the title box surrounding the table title region. The title arrangement direction recognition module is used to calculate the aspect ratio of the title box to determine the title arrangement direction of the table title; The text arrangement direction recognition module is used to detect all text detection boxes in the table image through a text detection model, and to classify and predict the text arrangement direction of all text detection boxes by combining the text direction model with the title arrangement direction. The table image correction module is used to correct the table image using the text arrangement direction and all text detection boxes to obtain a corrected image; The table type matching module is used to identify the table title within the title box in the corrected image through a text recognition model, and match the table type from a predefined table type configuration file based on the table title; The keyword extraction module is used to extract keywords from the corrected image based on the table type; The table information extraction module is used to process the extracted keywords through a post-processing function to extract table information.

7. The information extraction system with adaptive table text boundaries as described in claim 6, characterized in that: In the title box localization module, the instance segmentation model adopts either the YOLO model or the Mask R-CNN model; In the title arrangement direction recognition module, the title arrangement direction is either horizontal or vertical. In the text alignment direction recognition module, the text detection model adopts either the DB text detection model or the DB++ text detection model.

8. The information extraction system with adaptive table text boundaries as described in claim 6, characterized in that: The table image correction module is specifically used for: By controlling the text arrangement direction and the average centerline slope of all text detection boxes, the table image is subjected to an affine transformation around the image center point to obtain a corrected image.

9. The information extraction system with adaptive table text boundaries as described in claim 6, characterized in that: In the table type matching module, the text recognition model adopts either the SVTR model or the C-RNN model; Each of the aforementioned table types corresponds to a combination of table elements; the table elements include at least a table title, table lines, text content, seal, fingerprint, and signature.

10. The information extraction system with adaptive table text boundaries as described in claim 6, characterized in that: The keyword extraction module is specifically used for: Based on the table type, keywords are extracted from the corrected image by combining direct extraction algorithm and relation extraction algorithm; In the table information extraction module, the post-processing function includes at least a field mapping function, a text content replacement function, a symbol cleanup function, and a standardization function.

Images