A method for identifying complex table structures that combines top-down and bottom-up approaches.

Abstract

This invention discloses a method for recognizing complex table structures, including top-down grid detection and bottom-up grid merging. The method consists of four parts: a backbone network, a segmentation network, RoIAlign, and a graph attention network. Table structure recognition is a core problem in document automation; existing table structure recognition methods generally suffer from poor performance with large tables and slow inference speeds. This invention extracts features from table images of arbitrary sizes using a convolutional neural network; the graph attention network makes grid merging efficient and simple; and the introduction of boundary separators effectively reduces the difficulty of model learning.

Description

Technical Field

[0001] This invention relates to the fields of document analysis and computer vision technology. In particular, it relates to a method for recognizing complex table structures that combines top-down and bottom-up approaches. Background Technology

[0002] Table structure recognition is a challenging task due to the diversity of styles and structures. While table structure recognition has achieved excellent performance in recent years, driven by competitions related to table recognition, there are still areas for improvement, such as better handling tables with complex relationships.

[0003] Based on the different granularities chosen at the beginning of the problem processing, deep learning-based table structure recognition methods can be categorized into three types: top-down methods, bottom-up methods, and natural language processing methods. Top-down methods attempt to predict the row and column regions or row and column separators of the table, and then merge over-segmented regions with a row and column span greater than one using heuristic rules or sequence models. Although merging using sequence models is effective, it incurs significant computational costs and cannot be parallelized when the number of table cells is very large. Bottom-up methods start by detecting the position of text blocks or cells, and then recover the border relationships using heuristic rules or graph models. This method mostly ignores empty cells and is susceptible to local noise. Natural language processing methods define table structure recognition as a labeled sequence recognition problem. They attempt to directly translate table images into HTML or LaTeX code using recurrent neural networks or Transformers as language models. Due to the error accumulation inherent in autoregressive models, these methods often suffer from regression border drift, requiring complex post-processing for border matching. Furthermore, due to the limitations of linear layers in the network, the model can only accept inputs of a fixed size, and excessive downsampling leads to poor recognition performance for large tables.

[0004] In summary, existing table structure recognition methods still need improvement in the following aspects: 1. Inaccurate cell positioning; 2. Inability to balance speed and performance. Simple heuristic rules cannot handle complex merging relationships, while sequence models, although able to identify merging relationships well, have high computational complexity; 3. Poor performance in large tables. Summary of the Invention

[0005] This invention addresses the shortcomings of existing table structure recognition methods, such as inaccurate cell positioning, high computational cost of cell merging methods, and poor performance in recognizing large tables. It designs an effective method to identify table structures with complex straddling relationships. The invention mainly comprises two parts: 1. Top-down: Decomposing the table image into basic grid cells, which are divided by row and column separators predicted by a splitting model; 2. Bottom-up: A graph merging model predicts the merging relationships between adjacent grid cells in the horizontal and vertical directions. The specific steps are as follows:

[0006] (1) Image size processing. In this invention, the longest side of the input table image is limited to 842 pixels, which is the length of A4 paper pixels with a resolution of 72 pixels / inch.

[0007] (2) Obtain the feature representation of the image. The table image with a size of 3×H×W is fed into a fully convolutional network for feature extraction, and a three-dimensional tensor with a shape of C×H×W is obtained as the feature representation of the table image.

[0008] (3) Top-down grid detection. This invention introduces multi-branch receptive field blocks and spatial attention mechanisms into grid detection. Multi-branch receptive field blocks enhance the network's feature extraction capabilities by simulating the receptive field of human vision, while the spatial attention mechanism is used to learn the spatial correlation of features, thereby improving the accuracy of grid detection. Specifically, it can be further divided into the following three steps:

[0009] (3.1) Row separator detection. First, the three-dimensional tensor obtained in step (2) is fed into the multi-branch receptive field block. Then, the spatial attention coefficient matrix is ​​calculated for the output feature map, and the output feature map is weighted using the attention matrix. Finally, in order to transform the two-dimensional problem into a one-dimensional problem, the weighted feature map is subjected to projection pooling and activation to obtain the row separator prediction vector, denoted as H×1.

[0010] (3.2) Column separator detection. Except for projective pooling in the column dimension, the rest is the same as (3.1). Here, the column separator prediction vector is denoted as 1×W.

[0011] (3.3) Post-processing. The predicted vectors of row and column separators are binarized and some lightweight post-processing is performed to obtain M row separator segments and N column separator segments, thereby dividing the table image into an M×N grid.

[0012] (4) Bottom-up Grid Merging Relationship Prediction. This invention proposes a grid merging relationship prediction method based on graph attention networks. This method uses grid cells as vertices, samples edges between adjacent grid cells in the horizontal and vertical directions, and uses several graph attention blocks to enable information interaction between vertices, edges, and other elements, significantly improving the accuracy of merging relationship prediction. It can be further divided into the following five steps:

[0013] (4.1) Construction of the initial graph. An undirected graph G = (V, E) is constructed on the mesh output by the split model, where each vertex in V represents a basic mesh cell, and E is the set of edges between vertices. Except for boundary vertices, each vertex has 4 adjacent vertices in the horizontal and vertical directions. If two adjacent mesh cells need to be merged, the value of the corresponding edge is 1; otherwise, it is 0.

[0014] (4.2) Obtain the features of vertices and edges. For each vertex and edge, use RoI Align to extract a fixed-size R×R feature map from the feature representation of the table image obtained in step (2) as the visual feature of the vertex and edge, with a shape of C×3×3.

[0015] (4.3) Linear transformation. The C×3×3 three-dimensional feature map is flattened into a one-dimensional feature vector and then subjected to a linear transformation to reduce the dimension of the feature vector.

[0016] (4.4) Relationship prediction. The feature vectors of the edges of the nodes obtained in step (4.3) are fed into the graph attention network to classify the edges and obtain the predicted edge vectors, denoted as |E|×1.

[0017] (4.5) Post-processing. The edge prediction vectors are binarized and combined with the grid prediction results to recover the complete table structure.

[0018] (5) Table structure recognition test. Calculate the accuracy, recall, and F1 score between the Ground Truth and the predicted table structure.

[0019] Compared with existing table structure identification methods, the advantages of this invention are:

[0020] (1) This invention divides table structure recognition into two sub-tasks: top-down grid detection and bottom-up grid merging. For the first time, it treats the grid merging problem as an edge prediction problem of a graph. Compared to using graph models on text blocks or cells, this method reduces the total number of edges from O(KMN) to O(2MN-MN), significantly reducing computational complexity and accurately locating empty cells. Furthermore, compared to sequence models, this invention does not limit the size of the input table image, exhibiting good recognition accuracy even for large tables, and offering greater flexibility.

[0021] (2) This invention introduces multi-branch receptive field blocks and spatial attention mechanism into grid detection, guiding the network to focus on key identification regions in the table image, which effectively improves the accuracy of grid detection.

[0022] (3) In the delimiter detection stage, this invention considers both the boundary delimiter between the table and the outside world and the content delimiter between rows or columns. Since boundary delimiters and content delimiters have similar characteristics, the addition of boundary delimiters can reduce the learning difficulty of the model. Attached Figure Description

[0023] Figure 1 Table structure recognition framework diagram

[0024] Figure 2 Detailed diagram of separator detection

[0025] Figure 3 Initial graph construction Detailed Implementation

[0026] 1. Image preprocessing

[0027] This invention can accept table images of any size as input; however, for computational efficiency, while maintaining the same aspect ratio, the longest side of the input table image is limited to 842 pixels, which is the length of an A4 sheet of paper with a resolution of 72 pixels per inch. Furthermore, the input image is normalized.

[0028] 2. Obtaining the feature representation of the image

[0029] The preprocessed image is fed into a fully convolutional neural network pre-trained on ImageNet for feature extraction. The feature map with the same size as the original image after 32-fold upsampling is taken as the feature representation of the image, denoted as P.

[0030] 3. Separator detection

[0031] Taking line separators as an example, this article details the process of separator detection.

[0032] First, the feature vectors obtained in step 2 are fed into the multi-branch receptive field block.

[0033] Then, the attention coefficients are calculated and the features are weighted and summed:

[0034] a ij =conv(concat(average(e i，j，c ), max(e i，j，c ))).#(1)

[0035] a ij =sigmoid(a ij #(2)

[0036] e i，j，c =a ij *e i，j，c #(3)

[0037] Next, row projection pooling is performed on the weighted feature map:

[0038]

[0039]

[0040] Finally, activation and binarization are performed. Activation values ​​above the threshold are treated as separator pixels and set to 1.

[0041]

[0042]

[0043]

[0044] 4. Post-processing

[0045] Consecutive delimiter pixels are grouped together to obtain several delimiter segments. To eliminate false positives and oversegmentation issues during model prediction, we add some lightweight post-processing to the model. The first is to delete delimiter segments with a width less than 2, and the second is to connect two delimiter segments with a spacing of less than 5.

[0046] 5. Construction of the initial graph

[0047] For a table with M rows and N columns, there are |V| = M * N vertices and |E| = 2M * N - (M + N) edges. The vertices are numbered sequentially from left to right and from top to bottom, and the horizontal and vertical edges are numbered according to the vertex order:

[0048] V = {n0, n1, ..., n} i , ..., n |V|-1}.#(9)

[0049] E = {e0, e1, ..., e} |E|-1}.#(10)

[0050]

[0051] Where V and E represent the vertex set and edge set respectively, incidence is used to record the relationship between vertices and edges, and inc i，j This represents the relationship between the i-th vertex and the j-th edge.

[0052] 6. Extract features from vertices and edges

[0053] Based on the table image feature representation P obtained in step 2, RoIAlign is used to extract R×R fixed-size alignment features for vertices and edges, denoted as node_features and edge_features respectively:

[0054] rois_features = RoIAlign R×R (P, rois).#(12)

[0055] Where P is the tabular image feature representation extracted by the backbone network, and node_rois and edge_rois are the sets of regions of interest composed of all vertices and edges, respectively.

[0056] 7. Linear Transformation

[0057] Transform the feature maps of vertices and edges into the form of feature vectors required by the graph network:

[0058] x = Flatten(x).#(13)

[0059] x=ReLU(xW1+b1)W2+b2.#(14)

[0060] 8. Information Interaction

[0061] To establish local dependencies between adjacent vertices, we compute edge-to-vertex graph attention and vertex-to-edge graph attention to aggregate and update vertex and edge features:

[0062] nodes=Edge2VertexAtten(nodes, edges, incidence).#(15)

[0063] edges=Vertex2EdgeAtten(nodes, edges, incidence T #(16)

[0064] 9. Relationship Prediction

[0065] Based on the edge feature vectors (edges) obtained in the previous step, we predict the relationships between vertices, i.e., classify the edges. Specifically:

[0066] P edges =sigmoid(Linear(edges)).#(17)

[0067] 10. Post-processing

[0068] After merging, L-shaped cells and broken connections may occur. To address these issues, we implemented some simple post-processing methods, including: (1) when an L-shape exists in the merged cell, deleting the edge with the lowest score in the illegal cell until no L-shaped cells remain; (2) when most adjacent grid cells in a row (after the third row) have edges, adding merged edges between the broken grid cells in that row.

[0069] 11. Table Structure Recognition Test

[0070] The performance of the model in table structure recognition is evaluated using adjacency relationships. First, the table's Ground Truth and the recognition results are flattened into a one-dimensional list of paired adjacency relationships between cells. Then, precision, recall, and F1 score are calculated by comparing the two lists. The specific calculation method is as follows:

[0071]

[0072]

[0073]

Claims

1. A method for identifying complex table structures, comprising top-down grid detection and bottom-up grid merging, characterized in that, Includes the following steps: (1) Image preprocessing; (2) Obtain the feature representation of the image: The preprocessed image is fed into a fully convolutional neural network pre-trained on ImageNet for feature extraction. The feature map with the same size as the original image after 32 times upsampling is taken as the feature representation of the image, denoted as P; (3) Separator detection: First, the feature vector obtained in step (2) is sent into the multi-branch receptive field block; then, the attention coefficient is calculated and the features are weighted and summed; next, row projection pooling is performed on the weighted feature map; finally, activation and binarization are performed, and activation values ​​above the threshold are regarded as separators and set to 1; (4) Post-processing: Group consecutive separator pixels together to obtain several separator segments; (5) Initial graph construction: For an M-row N-column table, there are |V|=M*N vertices and |E|=2M*N-(M+N) edges. The vertices are numbered from left to right and from top to bottom, and the horizontal and vertical edges are numbered according to the vertex order: Where V and E represent the vertex set and edge set respectively, incidence is used to record the relationship between vertices and edges, and inc i,j This represents the relationship between the i-th vertex and the j-th edge; (6) Extracting vertex and edge features: Based on the table image feature representation P obtained in step (2), use RoIAlign to extract R×R fixed-size alignment features for vertices and edges respectively, denoted as node_features and edge_features: Where p is the table image feature representation extracted by the backbone network, and node_rois and edge_rois are the sets of regions of interest composed of all vertices and edges, respectively; (7) Linear transformation: Transforms the feature maps of vertices and edges into the form of feature vectors required by the graph network: (8) Information interaction: Calculate edge-to-vertex graph attention and vertex-to-edge graph attention to aggregate and update vertex and edge features: (9) Relationship prediction: Based on the edge feature vectors obtained in the previous step, the relationship between vertices is predicted, i.e., edge classification, specifically: (10) Post-processing: When there is an L-shape in the merged cell, delete the edge with the lowest score in the illegal cell until there are no L-shape cells; when most adjacent grid cells in a row have edges, add merged edges between the broken grid cells in that row.

2. The table structure recognition method as described in claim 1, characterized in that... Multi-branch receptive fields and attention mechanisms.

3. The table structure recognition method as described in claim 1, characterized in that... The delimiters include content delimiters and boundary delimiters.

4. The table structure recognition method as described in claim 1, characterized in that... The mesh merging method described is modeled using a graph neural network.

Images