A complex table recognition method and system

Abstract

This invention provides a method and system for complex table recognition, relating to the field of computer vision. The method includes: recognizing table outlines in a table image using multiple detection methods to obtain multiple recognition results; fusing the recognition results based on preset rules to obtain table bounding boxes; detecting key dividing elements capable of segmenting table content using multiple image processing methods; dividing the table into one or more block regions by combining the key dividing elements obtained from multiple image processing methods; performing independent and parallel analysis on each block region to obtain all cells; recognizing the text content of cells within each block region using an OCR algorithm to obtain the text content; establishing table hierarchical structure information based on the relative positions of the block regions; and reorganizing the text content according to the table hierarchical structure information to obtain the table recognition result. This method can accurately recognize and reconstruct tables containing arbitrarily complex merged cells spanning rows and columns.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and more specifically, to a method and system for recognizing complex tables. Background Technology

[0002] In various industries and daily life, there are numerous charts and graphs containing important data from different sectors. Extracting and reconstructing one or more complex, large tables from these files is often necessary. Existing methods for extracting and reconstructing complex tables often employ a bottom-up approach based on image processing, using edge detection and other methods to identify the intersections of row and column lines. However, this method relies heavily on the table's lines. When the lines in the chart are unclear or broken, errors can occur during extraction and reconstruction, ultimately leading to failure. Furthermore, it can misidentify merged complex cells, resulting in incorrect cell segmentation or structural reconstruction failure.

[0003] Therefore, improving the accuracy and robustness of table extraction and reconstruction for complex tables is an urgent problem to be solved. Summary of the Invention

[0004] This invention aims to solve at least one of the technical problems existing in the prior art or related technologies, and discloses a method and system for complex table recognition. First, it understands the overall layout of the table from a macro perspective, and then deconstructs it layer by layer down to the finest-grained cell, which greatly improves the accuracy and robustness of table extraction and reconstruction.

[0005] The first aspect of this invention discloses a method for recognizing complex tables, comprising: macroscopic layout analysis: recognizing the table outline in a table image based on multiple detection methods to obtain multiple recognition results, fusing the recognition results based on preset rules to obtain a table bounding box; main structure recognition: within the table bounding box, detecting main dividing elements capable of segmenting the table content based on multiple image processing methods, and dividing the table into one or more block regions by combining the main dividing elements obtained from multiple image processing methods, wherein the main dividing elements are dividing lines or visual blank bands; microscopic recognition: performing independent and parallel analysis and processing on the block regions to obtain all cells, and recognizing the text content of the cells in each block region using an OCR algorithm to obtain the text content; hierarchical reorganization: establishing table hierarchical structure information based on the relative positions of the block regions, and reorganizing the text content according to the table hierarchical structure information to obtain the table recognition result.

[0006] In this technical solution, the present invention is mainly used to solve the problem of recognizing tables with no lines, broken lines, or irregular layouts. The present invention proposes a "top-down" technical architecture. "Top-down" refers to a structural decomposition, while existing technologies first detect all underlying physical components (lines) and then reconstruct the structure through their geometric relationships (intersections). Although both technical approaches involve the process of "locating the table first, then processing its internal structure," the "top-down" approach of the present invention has a deeper meaning, reflected in the layer-by-layer decomposition of the table's logical structure, rather than merely the identification of physical components.

[0007] According to the complex table recognition method disclosed in this invention, preferably, the macro-layout analysis step specifically includes:

[0008] A visual feature detection method based on the DocLayout-YOLO model is used to identify table images and generate the first candidate bounding box.

[0009] A text layout detection method based on the TDeLTA model is used to identify table images and generate second candidate bounding boxes.

[0010] The table bounding box is obtained by fusing the first candidate bounding box and the second candidate bounding box based on preset rules. The preset rules include:

[0011] When the overlap between the first candidate bounding box and the second candidate bounding box reaches a threshold, the union of these two bounding boxes is taken as the table bounding box.

[0012] If only the DocLayout-YOLO model or only the TDeLTA model detects a certain region, the decision to accept the result is based on their respective confidence scores.

[0013] If the recognition results of the two models conflict, the recognition result of the TDeLTA model shall be adopted first.

[0014] According to the complex table recognition method disclosed in this invention, preferably, the main structure recognition step specifically includes:

[0015] Image processing method based on projection contour analysis: Within the table bounding box, the image is converted into a binary image; along the X-axis of the image, the pixel values ​​of each column are summed, and in the resulting vertical projection histogram, the trough regions correspond to the main branches in the vertical direction; along the Y-axis of the image, the pixel values ​​of each row are summed, and in the resulting horizontal projection histogram, the trough regions correspond to the main branches in the horizontal direction; by setting a trough threshold, small gaps inside the cells are filtered out, while the main branch elements that run through most of the table area are retained;

[0016] Image processing method for line detection based on Hough transform: Within the bounding box of a table, the Hough transform algorithm is applied to detect long line segments. By setting a length threshold to filter lines, only the long line segments that constitute the main frame of the table are retained as the main dividing elements.

[0017] Image processing method based on blank area analysis of text block layout: Obtain the bounding box of the text block, project these text blocks in the horizontal and vertical directions; determine the main dividing elements based on the gaps in the obtained projection histogram.

[0018] Combining the main segmentation elements obtained by the above three image processing methods: first, Hough transform is used to find all clear long line segments; then, in the areas where the long line segments cannot be completely segmented, projection contour analysis and text block layout analysis are used to find visual blank segmentation bands; finally, the coordinates of all identified main segmentation elements constitute the first coarse segmentation of the original table, and the segmented areas are block regions.

[0019] According to the complex table recognition method disclosed in this invention, preferably, the microscopic recognition step specifically includes:

[0020] Preliminary internal element detection: Within the block region, text block detection is performed first, followed by candidate cell detection, which involves running an algorithm based on contour or connected component analysis to find all independent content blocks within the region that could be cells.

[0021] Grid alignment and distribution analysis: After obtaining the center point coordinates and bounding boxes of all elements within the block area, statistical analysis is performed; the X coordinates of all element center points are analyzed. If the X coordinates can be clearly clustered into a few vertical lines, it indicates that these elements are aligned in the vertical direction, forming columns; the Y coordinates of all element center points are analyzed. If the Y coordinates can be clustered into a few horizontal lines, it indicates that rows are formed.

[0022] Regularity judgment: If all elements within a block region exhibit clustering characteristics in both the X and Y directions, and all elements have the same size, then the block region is considered structurally regular. If one or more of the following conditions occur, it is considered an irregular block region: the coordinate distribution of internal elements is disordered and cannot form clear row and column clusters; there are one or a few elements in the region whose size is much larger than other elements; there are large areas of blank space at the expected grid location, without any text blocks or content.

[0023] According to the complex table recognition method disclosed in this invention, preferably, the microscopic recognition step further includes:

[0024] For the block area that is determined to be regular, virtual grid lines are drawn based on the aligned row and column coordinates to divide the cells, and OCR recognition is performed on each divided cell.

[0025] For regions that are determined to be irregular, they are treated as indivisible cells, and OCR recognition is performed on these complete regions.

[0026] According to the complex table recognition method disclosed in this invention, preferably, the hierarchical restructuring step specifically includes:

[0027] Establish a global grid coordinate system: Obtain the coordinate information of the table bounding box and cells, and convert the local coordinates of the cells into global coordinates;

[0028] Global virtual grid construction: Based on global coordinates, extract the Y coordinates of the top and bottom edges and the X coordinates of the left and right edges of all cells, sort and remove duplicates, and generate virtual grid lines covering the entire table so that all cells have global coordinates based on the same origin and have a unified global virtual grid.

[0029] Cell logical mapping and attribute calculation: First, logical position mapping is performed, that is, for each cell with global coordinates, the relative position of each cell with global virtual grid lines is calculated to determine the starting row number and starting column number of the cell; then cross-dimensional attribute calculation is performed, that is, the number of grid lines crossed by the bottom and right boundaries of the cell is calculated to determine the number of rows and columns spanned by the cell.

[0030] Block area information inheritance: Each cell inherits the attribute information of its block area so that it can be used to distinguish the functional area to which the cell belongs or to perform differentiated processing on the cells in subsequent processing;

[0031] Occupy Map Initialization: Create a two-dimensional array with the same dimensions as the logical number of rows and columns of the global virtual grid, and initialize all elements to empty;

[0032] Priority processing: The cell list is processed in descending order of logical area, with priority given to merged cells. Merged cells are ensured to declare their occupied grid area first to avoid logical conflicts.

[0033] Occupation Declaration and Conflict Detection: For the currently processed cell, the logical grid area it covers is marked as a self-reference in a two-dimensional array. If the target area is already occupied, the conflict resolution mechanism is triggered. Finally, a fully filled and conflict-free occupation map is formed, which is used to reflect the ownership of each logical grid position.

[0034] Generation and serialization of table hierarchy tree: First, traverse the mapping table to generate the final hierarchical data structure; then create nodes. The root node represents the entire table, including the total number of rows and columns. Row nodes are the direct children of the root node and represent each row of the table. Cell nodes represent the children of row nodes.

[0035] Tree structure construction: Add cell nodes under their corresponding row nodes; cells spanning multiple rows are created only once in their starting row;

[0036] The above steps form a complete table hierarchy structure. Filling the recognized text content into the corresponding cells completes the table reconstruction, yielding the table recognition result.

[0037] According to the complex table recognition method disclosed in this invention, preferably, the conflict resolution mechanism specifically includes:

[0038] When a conflict is detected, the conflict type is first precisely classified:

[0039] Boundary overlap conflict: The conflict occurs at the adjacent boundaries of two different block areas, where the cell bounding boxes overlap.

[0040] Hierarchical conflict: Two cells overlap but belong to different block areas, and there is a clear containment or being contained relationship;

[0041] Conflict recognition error: The same physical location is identified as two separate cells, usually due to OCR or line detection errors;

[0042] For cases of overlapping boundaries: adjustments are made based on the segmentation priority of the block region, i.e., the main content region has a higher priority than the edge segmentation region; the virtual grid line position is finely adjusted to shift towards the lower priority block region to eliminate the overlap;

[0043] For cases of hierarchical conflicts: Analyze the hierarchical inclusion relationship of the block regions. If the block regions of two conflicting cells have a clear inclusion relationship, then enter the hierarchical arbitration, that is, check whether the block region of one cell completely contains the block region of another cell, and handle it according to the outer block priority principle or the nearest common ancestor block principle.

[0044] In cases of identification errors and conflicts: when hierarchical arbitration cannot make a clear decision, it indicates that the conflict is caused by identification errors. In this case, the confidence score generated during the identification of the main structure is introduced. Cells with high confidence scores are retained, and cells with low confidence scores are marked as pending review or directly removed. At the same time, this conflict event is recorded for subsequent model optimization.

[0045] According to the complex table recognition method disclosed in this invention, preferably, the outer block priority principle specifically includes: if the conflict is due to the accidental expansion of a cell in the inner block region into the outer block region, then the boundary of the inner cell is trimmed so that it is strictly confined within its own block region; the nearest common ancestor block principle specifically includes: if two conflicting block regions have the same parent block, then backtrack to their parent block region and perform hierarchical arbitration according to the layout rules within the parent block.

[0046] A second aspect of the present invention discloses a complex table recognition system, comprising: a memory for storing program instructions; and a processor for calling the program instructions stored in the memory to implement the complex table recognition method as described in any of the above technical solutions.

[0047] A third aspect of the present invention discloses a computer-readable storage medium storing program code for implementing a complex table recognition method as described in any of the above technical solutions.

[0048] The beneficial effects of the present invention include at least the following:

[0049] 1. A fundamental improvement in robustness and versatility: This invention eliminates the reliance on perfect row and column line intersections and, by identifying “main branch dividers” (including lines and blank bands), can effectively handle complex situations that are difficult for existing technologies to deal with, such as infinite tables, broken lines, and smudges.

[0050] 2. Precise analysis of complex structures: By introducing the innovative intermediate concept of "block region" and the "top-down" deconstruction idea, this invention can accurately identify and reconstruct tables containing arbitrarily complex merged cells across rows and columns, solving the core pain points of existing technologies.

[0051] 3. Higher reconstruction accuracy: The reconstruction method based on hierarchical context information ensures that the final generated table structure is logically consistent with the original document, avoiding global structural disorder caused by local recognition errors.

[0052] 4. Methodological advancement: This invention represents a paradigm shift from the traditional "bottom-up" approach based on micro-feature construction to the modern "top-down" approach based on macro-structure understanding, demonstrating significant advancement and originality in the field of table recognition. Attached Figure Description

[0053] Figure 1 A flowchart illustrating a complex table recognition method according to an embodiment of the present invention is shown.

[0054] Figure 2 A schematic block diagram of a complex table recognition system according to an embodiment of the present invention is shown. Detailed Implementation

[0055] To better understand the above-described objects, features, and advantages of the present invention, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Many specific details are set forth in the following description to provide a thorough understanding of the invention; however, the invention may be practiced in other ways different from those described herein, and therefore, the invention is not limited to the specific embodiments disclosed below.

[0056] Key term definitions:

[0057] Edge extraction: an important technique in image processing, mainly used to identify and locate regions in an image that show significant changes in brightness or color.

[0058] Connected component analysis: Connected component analysis is a common image processing operation. It is typically used for binary images to identify and label adjacent pixels with the same pixel value.

[0059] OCR stands for Optical Character Recognition. It involves scanning text on paper documents and converting it into a computer-programmable digital text format. OCR technology uses image processing and pattern recognition algorithms to identify the shape and features of characters, thereby enabling the automatic recognition and input of printed and handwritten text.

[0060] Hierarchical Tree: A table's hierarchical tree is a data structure that describes the containment and dependency relationships between cells in a table, defining the table's physical structure and logical hierarchy. A simple grid table without merged cells has a flat and simple hierarchical tree. A complex table with multi-level headers, however, has a deep and branching hierarchical tree.

[0061] Existing technologies, after identifying the table outline, aim to detect all horizontal and vertical lines within the entire table. Instead of a "break-down" approach, they perform a single line detection and intersection calculation for the entire table. This method heavily relies on clear, continuous table lines. For tables without lines, the method fails due to the lack of lines; for merged cells, the method produces serious errors due to missing intersections caused by line "interruptions."

[0062] Based on the above considerations, embodiments of the present invention disclose a complex table recognition method for multi-pass hierarchical complex table deconstruction and reconstruction. The core innovation of the present invention lies in proposing a "top-down" multi-pass hierarchical table processing paradigm. Unlike existing technologies that construct tables from micro-intersections, the present invention first understands the overall layout of the table from a macro perspective, and then deconstructs it layer by layer downwards until the finest-grained cells. The present invention deepens the understanding of the table structure through three levels of analysis: macro-layout analysis, backbone structure recognition, and micro-cell recognition. The present invention proposes a novel intermediate-level abstract concept, "block region": by recognizing the backbone dividers of the table (which can be physical lines or visual blank areas), the complex table is creatively decomposed into multiple easier-to-process, relatively simple "block regions". The present invention also proposes a precise reconstruction method based on a hierarchical tree: after completing the micro-recognition, using the hierarchical structure information established by the previous analysis, all cells are logically and completely reconstructed, perfectly restoring the complex merged cell relationships.

[0063] like Figure 1 As shown, according to an embodiment of the present invention, a method for recognizing complex tables is disclosed, comprising: inputting an image containing a complex table; first pass: macro layout analysis, used to locate the overall outer contour of the table; second pass: backbone structure recognition, used to deconstruct the table into a set of "block regions"; third pass: micro recognition, used to segment all cells and recognize the cell content; hierarchical reorganization: reorganizing according to the hierarchical context established in the first two passes; finally, outputting structured table data.

[0064] Specifically, the steps of the first round of macro-level layout analysis mentioned above include:

[0065] The system receives an input image containing a table. At the page-wide level, it first identifies and locates the table's overall bounding box, separating it from other non-table elements on the page (such as headings, paragraphs, and footnotes). This method can quickly and accurately separate a table as a whole object from the page without concern for its internal details. This invention employs an innovative "visual-text" dual-channel hybrid detection method. This method utilizes two models with different mechanisms in parallel and fuses their advantages to achieve robustness and accuracy unattainable by a single model. Instead of relying on a single detection logic, this method activates two dedicated detection channels in parallel:

[0066] The Visual Feature-Based Detection (DocLayout-YOLO) channel employs DocLayout-YOLO, an object detection model trained specifically on document layout elements using the YOLOv10 architecture. Unlike general YOLO models, DocLayout-YOLO's training dataset includes not only natural images but also a vast number of document images, enabling a deeper understanding of visual patterns within documents (such as lines, regular block-like content, headers, and footers). The model takes the pixels of the entire page image as input and its task is to identify and output bounding boxes for the "table" category. This channel performs well in detecting bounded tables and tables with clear visual boundaries, quickly capturing strong visual structural signals.

[0067] The Text-Based Detection (TDeLTA) channel employs the TDeLTA (Table Detection method based on Learning Text Arrangement) algorithm. The core idea of ​​TDeLTA is that a table is essentially a group of text arranged according to row and column rules. Therefore, this method does not directly analyze image pixels, but rather analyzes the spatial arrangement pattern of all text blocks on the page to locate the table. The input is the position coordinates of all text blocks. A sequence encoder and attention module model the arrangement relationship of these text blocks to determine which groups of text blocks constitute a table. This channel does not rely on visual lines, thus exhibiting good robustness to low-quality scans with non-linear tables and blurred or broken table lines.

[0068] Dual-channel result fusion and final bounding box determination: Each channel generates a set of candidate table bounding boxes. Finally, a decision fusion module merges these results according to preset rules to generate the final table outline.

[0069] If the bounding boxes output by DocLayout-YOLO and TDeLTA have a high degree of overlap, it is considered a high-confidence detection result, and the union of the two boxes can be used as the final bounding box. If only DocLayout-YOLO detects a region (which might be an empty table with borders but no text), or only TDeLTA detects a region (which might be an empty table), the system will decide whether to accept the result based on their respective confidence scores. In rare cases where the results of the two models conflict, TDeLTA's result can be prioritized because it has a stronger generalization ability to complex and irregular tables.

[0070] With this dual-channel hybrid detection method, the system can utilize both visual cues and text layout cues in the first pass, thus achieving better accuracy and robustness than any single method (especially traditional methods that rely on lines) in the initial step of locating table bounding boxes.

[0071] According to the above embodiments, further, for the extraction of text information, in order to support the above dual-channel detection, this method divides the extraction of text information into two stages:

[0072] The first pass of text information extraction involves obtaining the location information (i.e., bounding box coordinates) of all text blocks. These coordinates are the direct input to the TDeLTA model and will also serve as important data for the second and third passes of analysis. In the macro-analysis, only rapid text detection is performed, without full text content recognition (OCR). All regions on the page that may contain text are identified, and a bounding box is generated for each region. This step is executed in parallel with the visual analysis of DocLayout-YOLO.

[0073] Subsequent steps involve text extraction: complete, high-precision Open Content Recognition (OCR), which extracts specific text content from the text box, performed in the micro-recognition step. OCR is then applied to small, defined "block regions" that have been confirmed to belong to a specific cell. This "delayed processing, on-demand recognition" strategy improves the overall system's processing efficiency.

[0074] According to the above embodiment, the second step of identifying the main structure specifically includes:

[0075] Within the initially defined table outline, instead of searching for fine-grained lines for individual cells, the focus is on identifying the main dividing lines that divide the table into several key regions. These main dividing lines can be physical lines running through the entire table or most of its area, or visually significant blank bands. Based on these main dividing lines, the table is logically divided into one or more large "block regions." The core of this step is to break down a complex, potentially irregular table into several simpler, more regular sub-regions (i.e., "block regions"). This step avoids the vulnerabilities of existing techniques that directly handle microscopic details.

[0076] A hybrid analysis method is employed, combining image features and text block location information obtained in the first pass to find the "core segment". The hybrid analysis method includes: projection contour analysis based on image processing, line detection based on Hough transform, and blank area analysis based on text block layout.

[0077] Image processing methods based on projection profile analysis:

[0078] Within the bounding box of the table located in the first pass, the image is converted into a binary image. Vertical projection: The pixel values ​​of each column are summed along the X-axis of the image. In the resulting projected histogram, continuous "valley" areas with pixel sums close to zero correspond to the vertical main dividers in the table (whether physical lines or large blank areas). Horizontal projection: Similarly, the pixel values ​​of each row are summed along the Y-axis; the "valleys" in the histogram correspond to the horizontal main dividers. By setting a threshold (e.g., the depth and width of the valleys), small gaps within cells can be filtered out, retaining the significant dividers that run through most of the table.

[0079] Image processing methods based on Hough Transform for Line Detection:

[0080] This method is specifically designed for detecting physical separator lines in tables. Within the table's bounding box, the Hough transform algorithm is applied to detect long straight line segments. By setting a length threshold, short lines that might be part of a character can be filtered out, retaining only the long straight lines that form the main framework of the table.

[0081] Image processing methods based on text block layout whitespace analysis:

[0082] Using the bounding boxes of all text blocks acquired in the first pass, these text blocks are projected horizontally and vertically (similar to projection contour analysis, but operating on text boxes instead of pixels). Large gaps are then identified in the projection histogram of the text boxes; these gaps represent main structural divisions. Because this method is completely independent of visual lines, it is particularly effective for wireless tables.

[0083] Finally, the three methods described above are combined. First, Hough transform is used to find all definite physical long lines. Then, in the areas not completely divided by these long lines, projective contour analysis and text block layout analysis are used to find visual blank dividing bands. Ultimately, the coordinates of the identified main dividing lines (physical or blank) constitute the first "coarse segmentation" of the original table, and the segmented areas are "block regions".

[0084] According to the above embodiments, the third step of microscopic identification specifically includes:

[0085] For the second pass of the above: main structure recognition, each identified "block region" is analyzed and processed independently and in parallel: if the internal structure of a "block region" is regular (for example, a simple M*N grid), then efficient and conventional cell segmentation and OCR recognition algorithms are used for it. If a "block region" itself is an anomalous unit (for example, a large merged cell), then it is treated as an indivisible atomic unit for overall content recognition.

[0086] A method based on internal structural consistency verification is used to determine whether the "block region" is regular:

[0087] Preliminary Internal Element Detection: Within a "block region," a lightweight internal element detection is first run. This begins with text block detection, directly utilizing all text blocks that fall into the "block region" in the first pass; then, candidate cell detection is performed, running a fast algorithm based on contour or connected component analysis to find all independent content blocks within the region that could potentially be cells.

[0088] Grid alignment and distribution analysis: After obtaining the center point coordinates and bounding boxes of all internal elements (mainly text blocks), statistical analysis is performed.

[0089] Vertical alignment: Analyze the X coordinates of the center points of all elements. If these X coordinates can be clearly clustered onto a few vertical lines, it means that the elements are aligned vertically and form columns.

[0090] Horizontal alignment: Similarly, analyze the Y-coordinates of the center points of all elements. If the Y-coordinates can be clustered onto a few horizontal lines, it indicates that rows have been formed.

[0091] Based on the above method, determine whether a cell is regular:

[0092] If all elements within a "block region" exhibit strong clustering characteristics in both the X and Y directions, and all elements are roughly the same size, then this "block region" is considered to be structurally regular (e.g., a standard M*N grid).

[0093] If one or more of the following conditions are met, the structure is considered abnormal:

[0094] Alignment failed: The coordinates of the internal elements are disorganized and cannot form clear row and column clusters.

[0095] Significant size disparity: The area contains one or a few elements whose size is much larger than the other elements. This is most likely a merged cell, as it occupies the space of multiple standard cells.

[0096] Large areas of blank space: Large areas of blank space exist in the expected grid locations, without any text blocks or content.

[0097] For regions deemed "normal," conventional algorithms are applied directly, using virtual grid lines based on their aligned row and column coordinates for cell segmentation and OCR recognition. Regions deemed "structurally abnormal" are treated as indivisible atomic units. In the final reconstruction stage, this complete region is subjected to OCR once, extracting and reconstructing all its contents as a whole.

[0098] According to the above embodiments, the hierarchical reorganization step further includes:

[0099] Based on the table hierarchy information established in the first and second passes (i.e., the relative position of each "block region" in the overall table), all micro-identification results from the third pass (including regular cells and merged cells as atomic units) are precisely and logically reorganized. The result of the reorganization is a hierarchical data structure (such as nested lists or tree structures) capable of representing complex cross-row and cross-column relationships, thus completing the accurate reconstruction of the original table.

[0100] After completing the microscopic identification, a set of all block regions and their internal atomic cells is obtained. Then, using macroscopic layout and intra-block structure, the discrete identification results are merged into a globally unified and logically correct table hierarchy tree. This reorganization method specifically includes the following steps:

[0101] Global grid coordinate system and block region location fusion: Obtain macroscopic analysis results, i.e., the global bounding boxes of each block region, and microscopic identification results, i.e., the local coordinates and contents of all atomic cells within each block region. Then, establish a unified global coordinate system to convert the local coordinates within the block region into global coordinates. During coordinate transformation, for any atomic cell, its global coordinates are calculated through an affine transformation of the global bounding box of its respective block region.

[0102] Global Virtual Grid Construction: Aggregates the global bounding box information of all atomic cells. Extracts the Y-coordinates of the top and bottom edges and the X-coordinates of the left and right edges of all cells, sorts and removes duplicates, and generates virtual grid lines covering the entire table. All atomic cells have global coordinates based on the same origin and possess a unified global virtual grid.

[0103] Cell logical mapping and attribute calculation based on virtual grid: First, logical position mapping is performed, that is, for each atomic cell with global coordinates, its relative position with the global virtual grid lines is calculated to determine its starting row number and starting column number. Then, cross-dimensional calculation is performed, that is, the number of grid lines crossed by the bottom and right boundaries of the cell is calculated to determine its number of rows and columns spanned.

[0104] Block area information inheritance: Each cell inherits the attribute information of its corresponding block area. This information can be used in subsequent processing to distinguish different functional areas such as the table header and body, or to perform differentiated processing. Specifically, each atomic cell is assigned a complete five-tuple logical attribute: (start_row, start_col, rowspan, colspan, text_content), and associated with its block_id. Here, start_row represents the starting row index (starting from 0), start_col represents the starting column index (starting from 0), rowspan represents the number of cells merged vertically (default is 1), colspan represents the number of cells merged horizontally (default is 1), text_content represents the cell text content, and block_id represents the block area name.

[0105] To ensure logical consistency in the generated structure tree, the following strategies are employed for conflict resolution and hierarchy tree construction:

[0106] Occupy Map Initialization: Create a two-dimensional array with the same number of logical rows and columns as the global virtual grid, and initialize all elements to null.

[0107] Priority processing: The cell list is processed in descending order of logical area. This strategy prioritizes merged cells, ensuring they declare their occupied grid area first to avoid logical conflicts.

[0108] Occupation Declaration and Conflict Detection: For the currently processed cell, the logical grid region it covers is marked as a self-reference in a two-dimensional array. If the target region is already occupied, a conflict resolution mechanism is triggered. According to a preferred embodiment of this method, the block region segmentation information in the first step can be traced back to check whether the conflict occurs at the block region boundary, and arbitration is performed based on the hierarchical relationship of the block regions.

[0109] This results in a fully filled and conflict-free occupancy mapping table that reflects the ownership of each logical grid location.

[0110] When a conflict is detected, the conflict type is first precisely classified:

[0111] Boundary overlap conflict: The conflict occurs at the adjacent boundaries of two different block areas, where the cell bounding boxes overlap (usually ≤3 pixels).

[0112] Hierarchical relationship conflict: Two cells completely or mostly overlap and belong to different block areas, with a clear containment or being contained relationship.

[0113] Conflict recognition error: The same physical location is identified as two separate cells, usually due to OCR or line detection errors.

[0114] When arbitrating the hierarchical relationship of block region boundary conflicts, the first step is to check the relationship between the conflict location and the block region boundary. This involves obtaining the `block_id` of the cells involved in the conflict and querying the relative spatial relationship (e.g., top-bottom, left-right, containment) between the two block regions recorded in the first step. If the conflict occurs at the boundary of two adjacent block regions and the overlapping area is small (e.g., <5% of the cell area), it is considered a boundary overlap conflict. Then, adjustments are made based on the block region segmentation priority. Typically, the system assigns higher priority to the main content area than to the edge separation area. For example, if one cell comes from the "Table Body" block and the other from the "Decorative Line" block, the cell from the "Table Body" block is retained first. Overlap is eliminated by fine-tuning the position of the virtual grid lines (offsetting them 1-2 pixels towards the lower-priority block region).

[0115] To address hierarchical conflicts, the hierarchical containment relationship of block regions is analyzed. If the block regions of two conflicting cells have a clear containment relationship (such as an identified parent block and child block), then hierarchical arbitration is initiated. This involves checking whether the block region of one cell completely contains the block region of another cell, following the principle of "outer block priority" or "nearest common ancestor block".

[0116] Outer block priority: If a conflict occurs because a cell in an inner block region unexpectedly expands into an outer block region, the boundaries of the inner cell are trimmed to strictly confine it within its own block region. For example, if a cell in a header block extends down into a table body block, the cell is trimmed according to the bottom boundary of the header block.

[0117] The closest common ancestor block: If two conflicting blocks are siblings (share the same parent block), then the process backtracks to their parent block region and arbitrates according to the layout rules within the parent block. For example, within the parent block, a "top-to-bottom, left-to-right" priority is used.

[0118] When arbitration based on block regions cannot reach a clear decision, such as when conflicts arising from identification errors or hierarchical relationships cannot be definitively resolved, confidence scores generated during the main structure identification process are introduced (e.g., OCR recognition confidence, line detection intensity values; in the second step of main structure identification, when the system performs detailed cell and main structure identification on each "block region," its internal process also generates various types of confidence scores). Cells with high confidence are retained, while cells with low confidence are marked as "pending review" or directly removed. Simultaneously, this conflict event is recorded for subsequent model optimization by the system.

[0119] Table hierarchy tree generation and serialization: First, traverse the mapping table to generate the final hierarchical data structure. Then, create nodes. The root node represents the entire table and contains metadata such as the total number of rows and columns. Row nodes are the direct children of the root node and represent each row of the table. Cell nodes are the children of row nodes and are created only when traversing to the starting grid position of a cell.

[0120] Tree structure construction: Cell nodes are added under their corresponding row nodes. Multi-row cells are created only once in their starting row, and their `rowspan` and `colspan` attributes define their extent, logically forming a tree. The output is then serialized, and this hierarchical tree structure can be losslessly serialized into various industry-standard formats, including: HTMLTable: direct mapping, generating `rowspan` and `colspan` attributes for `td` tags based on cell attributes; JSON: generating a structured data object containing row and column structure, cell content, and merging relationships; Excel files: creating worksheets containing merged cells via a programming interface.

[0121] The final output is a hierarchical tree object that fully describes the logical structure and content information of the original table, along with its various serialization formats.

[0122] like Figure 2 As shown, according to another embodiment of the present invention, a complex table recognition system 200 is also disclosed, including: a memory 201 for storing program instructions; and a processor 202 for calling the program instructions stored in the memory to implement the complex table recognition method as described in the above embodiment.

[0123] According to another embodiment of the present invention, a computer-readable storage medium is also disclosed, which stores program code for implementing the complex table recognition method as described in the above embodiments.

[0124] In summary, this invention proposes a multi-pass hierarchical table processing paradigm:

[0125] 1) First pass: The system first identifies the outer border of the entire table.

[0126] 2) Second pass: The system searches for the main dividing lines within the table. It identifies the main column dividers that run through the table, as well as unbroken row dividers above and below merged cells. At this point, the large merged cell itself, along with the regular columns to its left and right, is recognized as a separate, large "block area." The other regular parts of the table are recognized as separate "block areas."

[0127] 3) Third pass: The system processes each "block region" separately. For regular block regions, it quickly segments and identifies cells. For the large block region containing merged cells, the system identifies it as an indivisible atomic unit and performs OCR on the entire block.

[0128] 4) Fourth pass (hierarchical reorganization): The system reorganizes the data based on the results of the second pass. It recognizes that the large merged cell logically spans multiple rows and is placed alongside other regular cells. Ultimately, it generates an accurate data structure that perfectly replicates the original table structure.

[0129] Through the above embodiments disclosed in this invention, tables containing arbitrarily complex merged cells spanning rows and columns can be accurately identified and reconstructed, solving the core pain points of the prior art.

[0130] All or part of the steps in the various methods of the above embodiments can be implemented by a program controlling the relevant hardware. The program can be stored in a readable storage medium, including read-only memory (ROM), random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically-erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, disk storage, magnetic tape storage, or any other readable medium that can be used to carry or store data.

[0131] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for recognizing complex tables, characterized in that, include: Macro layout analysis: Based on multiple detection methods, the table outline in the table image is identified, and multiple identification results are obtained. The identification results are fused based on preset rules to obtain the table bounding box; Main structure recognition: Within the bounding box of the table, the main dividing elements that can segment the table content are detected based on multiple image processing methods. The table is divided into one or more block regions by combining the main dividing elements obtained by multiple image processing methods. The main dividing elements are dividing lines or visual blank bands. Microscopic recognition: The block area is analyzed and processed independently and in parallel to obtain all cells. The text content of the cells in each block area is recognized by the OCR algorithm to obtain the text content. Hierarchical Reorganization: Based on the relative positions of the block regions, a table hierarchy structure is established, and the text content is reorganized according to the table hierarchy structure information to obtain the table recognition result; The steps of the macro-layout analysis specifically include: The visual feature detection method based on the DocLayout-YOLO model recognizes table images and generates first candidate bounding boxes. The DocLayout-YOLO model is an object detection model based on the YOLOv10 architecture and specifically trained for document layout elements. The text arrangement detection method based on the TDeLTA model is used to identify table images and generate second candidate bounding boxes. The TDeLTA model is a table detection method based on learning text arrangement model. The first candidate bounding box and the second candidate bounding box are fused based on preset rules to obtain a table bounding box, wherein the preset rules include: When the overlap between the first candidate bounding box and the second candidate bounding box reaches a threshold, the union of these two bounding boxes is taken as the table bounding box. If only the DocLayout-YOLO model or only the TDeLTA model detects a certain region, the decision to accept the result is based on their respective confidence scores. If the recognition results of the two models conflict, the recognition result of the TDeLTA model shall be adopted first. The steps for identifying the main structure specifically include: Image processing method based on projection contour analysis: Within the table bounding box, the image is converted into a binary image; along the X-axis of the image, the pixel values ​​of each column are summed, and in the resulting vertical projection histogram, the trough regions correspond to the main branches in the vertical direction; along the Y-axis of the image, the pixel values ​​of each row are summed, and in the resulting horizontal projection histogram, the trough regions correspond to the main branches in the horizontal direction; by setting a trough threshold, gaps inside the cells are filtered out, and the main branch elements that run through the table are retained; Image processing method for line detection based on Hough transform: Within the bounding box of a table, the Hough transform algorithm is applied to detect long line segments. By setting a length threshold to filter lines, only the long line segments that constitute the main frame of the table are retained as the main dividing elements. Image processing method based on blank area analysis of text block layout: Obtain the bounding box of the text block, project these text blocks in the horizontal and vertical directions; determine the main dividing element based on the gaps in the obtained projection histogram; The main segmentation elements obtained by combining image processing methods based on projection contour analysis, line detection based on Hough transform, and blank area analysis based on text block layout are as follows: First, Hough transform is used to find all clear long line segments; then, in the areas where the long line segments cannot be completely segmented, projection contour analysis and text block layout analysis are used to find visual blank segmentation bands; finally, the coordinates of all identified main segmentation elements constitute the first coarse segmentation of the original table, and the segmented areas are block regions.

2. The complex table recognition method according to claim 1, characterized in that, The steps of the microscopic identification specifically include: Preliminary internal element detection: Within the block region, text block detection is performed first, followed by candidate cell detection, which involves running an algorithm based on contour analysis or connected component analysis to identify all independent content blocks within the region. Grid alignment and distribution analysis: After obtaining the center point coordinates and bounding boxes of all elements within the block area, statistical analysis is performed; the X coordinates of all element center points are analyzed. If the X coordinates can be clearly clustered onto vertical lines, it indicates that these elements are aligned in the vertical direction, forming columns; the Y coordinates of all element center points are analyzed. If the Y coordinates can be clustered onto horizontal lines, it indicates that rows are formed. Regularity judgment: If all elements in a block region exhibit clustering characteristics in both the X and Y directions, and all elements have the same size, then the block region is considered to be structurally regular. If one or more of the following conditions occur, it is considered to be an irregular block region: the coordinate distribution of the internal elements is disordered and cannot form clear row and column clusters; there are elements in the region that are much larger than other elements; there are no text blocks or content at the expected grid positions.

3. The complex table recognition method according to claim 2, characterized in that, The microscopic identification steps also include: For the block area that is determined to be regular, virtual grid lines are drawn based on the aligned row and column coordinates to divide the cells, and OCR recognition is performed on each divided cell. For regions that are determined to be irregular, they are treated as indivisible cells, and OCR recognition is performed on these complete regions.

4. The complex table recognition method according to claim 1, characterized in that, The hierarchical reorganization steps specifically include: Establish a global grid coordinate system: Obtain the coordinate information of the table bounding box and the cell, and convert the local coordinates of the cell into global coordinates; Global virtual grid construction: Based on the global coordinates, extract the Y coordinates of the top and bottom edges and the X coordinates of the left and right edges of all cells, sort and remove duplicates, and generate virtual grid lines covering the entire table so that all cells have global coordinates based on the same origin and have a unified global virtual grid. Cell logical mapping and attribute calculation: First, logical position mapping is performed, that is, for each cell with global coordinates, the relative position of each cell with global virtual grid lines is calculated to determine the starting row number and starting column number of the cell; then cross-dimensional attribute calculation is performed, that is, the number of grid lines crossed by the bottom and right boundaries of the cell is calculated to determine the number of rows and columns spanned by the cell. Block area information inheritance: Each cell inherits the attribute information of its block area so that it can be used to distinguish the functional area to which the cell belongs or to perform differentiated processing on the cells in subsequent processing; Occupy Map Initialization: Create a two-dimensional array with the same dimensions as the logical number of rows and columns of the global virtual grid, and initialize all elements to empty; Priority processing: The cell list is processed in descending order of logical area, with priority given to merged cells. Merged cells are ensured to declare their occupied grid area first to avoid logical conflicts. Occupation Declaration and Conflict Detection: For the currently processed cell, the logical grid area it covers is marked as a self-reference in a two-dimensional array. If the target area is already occupied, the conflict resolution mechanism is triggered. Finally, a fully filled and conflict-free occupation map is formed, which is used to reflect the ownership of each logical grid position. Generation and serialization of table hierarchy tree: First, traverse the mapping table to generate the final hierarchical data structure; then create nodes. The root node represents the entire table, including the total number of rows and columns. Row nodes are the direct children of the root node and represent each row of the table. Cell nodes represent the children of row nodes. Tree structure construction: Add cell nodes under their corresponding row nodes; cells spanning multiple rows are created only once in their starting row; The above steps form a complete table hierarchy structure. Filling the recognized text content into the corresponding cells completes the table reconstruction, yielding the table recognition result.

5. The complex table recognition method according to claim 4, characterized in that, The conflict resolution mechanism specifically includes: When a conflict is detected, the conflict type is first precisely classified: Boundary overlap conflict: The conflict occurs at the adjacent boundaries of two different block areas, where the cell bounding boxes overlap. Hierarchical conflict: Two cells overlap but belong to different block areas, and there is an inclusive or contained relationship between them; Conflict identification error: The same physical location is identified as two separate cells; For the aforementioned boundary overlap conflict: adjustments are made based on the segmentation priority of the block region, i.e., the main content region has a higher priority than the edge segmentation region; the virtual grid line position is finely adjusted to shift towards the lower priority block region to eliminate the overlap; In the case of the aforementioned hierarchical relationship conflict: analyze the hierarchical inclusion relationship of the block regions. That is, if the block regions to which the two conflicting cells belong have a clear inclusion relationship, then enter the hierarchical arbitration, that is, check whether the block region of one cell completely contains the block region of another cell, and handle it according to the outer block priority principle or the nearest common ancestor block principle. Regarding the aforementioned identification error conflict: when the hierarchical arbitration cannot make a clear judgment, it indicates that the conflict is caused by an identification error. In this case, the confidence score generated during the main structure identification process is introduced, cells with high confidence are retained, and cells with low confidence are marked as pending review or directly removed.

6. The complex table recognition method according to claim 5, characterized in that, The outer block priority principle specifically includes: if the conflict is due to the accidental expansion of an inner block region's cell into an outer block region, then the boundary of the inner cell is trimmed to strictly confine it within its own block region; the nearest common ancestor block principle specifically includes: if two conflicting block regions have the same parent block, then backtrack to their parent block region and perform hierarchical arbitration according to the layout rules within the parent block.

7. A complex table recognition system, characterized in that, include: Memory, used to store program instructions; A processor is configured to invoke the program instructions stored in the memory to implement the complex table recognition method as described in any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores program code for implementing the complex table recognition method as described in any one of claims 1 to 6.

Images