Table identification method and system based on GNN and OCR system

Abstract

The invention belongs to the technical field of document processing, and discloses a table recognition method and system based on GNN and OCR systems, and the method comprises the steps: obtaining a to-be-recognized table image; extracting image visual features and text semantic features of the table image to be recognized, and fusing the extracted image visual features and text semantic features to obtain a fused feature map; inputting the fused feature map into a segmentation network constructed based on a CellPose model, and outputting a physical layout of the cells; and constructing a graph structure based on the physical layout of the cells, reasoning the same-row and same-column relationship among the cells by utilizing GNN, distributing row and column indexes through a union lookup set algorithm, identifying and merging the cells in combination with geometric rules, endowing semantic tags, and outputting a table identification result in a preset format. According to the technical scheme, the problems in the aspects of complex table structure recognition, wireless table processing and semantic understanding are solved, and the accuracy and the automation degree of table recognition are remarkably improved.

Description

Technical Field

[0001] This invention belongs to the field of document processing technology, and in particular relates to a table recognition method and system based on GNN and OCR systems. Background Technology

[0002] As digital transformation deepens, a large number of tables in paper documents and electronic images need to be automatically identified and converted into structured data for storage, retrieval, and analysis.

[0003] Traditional table recognition methods typically rely heavily on Optical Character Recognition (OCR) technology and rule-based post-processing. These methods first use an OCR engine to locate and recognize all text in an image, then infer the table outline using heuristic or line detection algorithms, and finally reconstruct the table structure based on the positional relationship between the text and the outline. However, these methods have significant limitations: First, their recognition performance is highly dependent on the accuracy of the OCR and the integrity of the table outline; the recognition rate drops sharply for incomplete, faulty, or tilted tables. Second, traditional methods separate text recognition from structural recognition, lacking an understanding of the table's global context and struggling to effectively handle complex merged cells. Finally, the output is usually a set of text and coordinates, lacking semantic understanding of the cell content, and thus failing to meet the deeper needs of downstream data analysis and system integration. Summary of the Invention

[0004] The purpose of this invention is to provide a table recognition method and system based on graph neural network (GNN) and OCR system to solve the problems existing in the prior art.

[0005] To achieve the above objectives, this invention provides a table recognition method based on GNN and OCR systems, comprising:

[0006] Obtain the image of the table to be recognized;

[0007] Extract the visual features and semantic features of the table image to be identified, and fuse the extracted visual features and semantic features to obtain a fused feature map;

[0008] The fused feature map is input into the segmentation model to predict the physical layout of the cells, wherein the segmentation model is built based on the CellPose model;

[0009] Based on the physical layout of cells, combined with graph neural networks, the row and column relationships between cells are inferred;

[0010] The inference results based on graph neural networks are post-processed, specifically including: assigning all cells to the correct row and column indices, and assigning semantic labels to each cell;

[0011] After post-processing, the table recognition results in a preset format are output.

[0012] Optionally, the process of acquiring the image visual features specifically includes:

[0013] The visual features of the table image to be identified are extracted using a Transformer-based visual model to obtain a visual feature map.

[0014] Optionally, the process of obtaining the text semantic features specifically includes:

[0015] Based on the OCR system, text detection and text recognition are performed on the table image to be recognized to obtain several text tokens, the text tokens including text content and bounding box coordinates;

[0016] The text content of each text token is converted into a text feature vector;

[0017] Generate a list of text tokens to extract the semantic features of the text.

[0018] Optionally, the process of obtaining the fused feature map specifically includes:

[0019] The bounding box of each text token is mapped onto the visual feature map, and the corresponding text feature vector is concatenated with the visual feature vector to obtain the fused feature map.

[0020] Optionally, inputting the fused feature map into the physical layout of the predicted cells in the segmentation model specifically includes:

[0021] The fused feature map is segmented based on the CellPose model to obtain the segmented cell instance mask;

[0022] Calculate the minimum bounding rectangle of each cell instance mask to obtain the cell coordinates, and generate an initial cell proposal based on the cell instance mask and coordinates.

[0023] Optionally, the inference of row and column relationships between cells using graph neural networks specifically includes:

[0024] A graph-structured data is constructed, with each cell as a node and each node corresponding to a feature vector, which includes visual and geometric features. The K nearest neighbor algorithm is used to construct edges, connecting each cell to its k spatial nearest neighbors based on the coordinates of the cell's center point.

[0025] The constructed graph structure data is input into a graph neural network to predict the row and column relationship between the two cells connected by each edge. The row and column relationship includes unrelated, same row, and same column.

[0026] Optionally, the inference results based on the graph neural network undergo post-processing, specifically including:

[0027] Obtain the set of edges related to the same row, find all rows using the disjoint-set data structure algorithm, sort the rows from top to bottom according to their average Y-coordinate, and assign row indices;

[0028] Obtain the set of edges with the same column relationship, find all columns using the disjoint-set data structure algorithm, sort the columns from left to right according to their average X-coordinate, and assign column indices;

[0029] Calculate the average height of all cells in each row and the average width of all cells in each column;

[0030] For each cell, if the physical size of the cell is greater than the average size of the corresponding row and column, then the current cell is a merged cell;

[0031] Based on the coordinate overlap, the text semantic features are matched to the corresponding cells;

[0032] The named entity recognition model is used to classify and predict the text content of each cell to obtain the semantic label corresponding to each cell.

[0033] The physical layout, row and column structure parsing, and semantic tags corresponding to each table are serialized to output structured table recognition results.

[0034] On the other hand, to achieve the above objectives, the present invention provides a table recognition system based on GNN and OCR systems, comprising:

[0035] The data acquisition module is used to acquire the image of the table to be recognized;

[0036] The multimodal feature processing module is used to extract the image visual features and text semantic features of the table image to be identified, and to fuse the extracted image visual features and text semantic features to obtain a fused feature map;

[0037] A cell detection module is used to input the fused feature map into the segmentation model to predict the physical layout of cells, wherein the segmentation model is built based on the CellPose model;

[0038] The relational reasoning module is used to infer the row and column relationships between cells based on the physical layout of the cells and in conjunction with a graph neural network.

[0039] The post-processing module is used to perform post-processing based on the inference results of the graph neural network. Specifically, it includes: assigning all cells to the correct row and column indices and assigning semantic labels to each cell.

[0040] The output module is used to output the table recognition results in a preset format.

[0041] The technical effects of this invention are as follows:

[0042] This invention utilizes multimodal feature fusion technology, enabling the model to incorporate textual semantic information while perceiving images. This allows it to effectively handle complex scenarios such as wireless tables and distorted tables, with recognition accuracy and robustness far exceeding traditional methods that rely on line detection.

[0043] This invention utilizes a CellPose-based instance segmentation model to generate cell proposals, which can handle closely adjacent cells and avoid complex line prediction and post-processing. At the same time, GNN gathers neighbor context information through a message passing mechanism, which can accurately infer the logical structure of complex cell tables, further improving the recognition accuracy and robustness in complex scenarios.

[0044] The final recognition result output by this invention includes structured data containing physical coordinates, row and column indices, merging information, and semantic tags, which can be directly called by databases, data analysis software, or business systems, thereby improving data availability and automation. Attached Figure Description

[0045] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0046] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0047] Figure 1 This is a flowchart of the table recognition process in an embodiment of the present invention;

[0048] Figure 2 This is a schematic diagram of the system structure in an embodiment of the present invention. Detailed Implementation

[0049] Various exemplary embodiments of the present invention will now be described in detail. This detailed description should not be considered as a limitation of the present invention, but rather as a more detailed description of certain aspects, features, and embodiments of the present invention.

[0050] It should be understood that the terminology used in this invention is merely for describing particular embodiments and is not intended to limit the invention. Furthermore, with respect to numerical ranges in this invention, it should be understood that each intermediate value between the upper and lower limits of the range is also specifically disclosed. Every smaller range between any stated value or intermediate value within a stated range, and any other stated value or intermediate value within said range, is also included in this invention. The upper and lower limits of these smaller ranges may be independently included or excluded from the range.

[0051] Various modifications and variations can be made to the specific embodiments described in this specification without departing from the scope or spirit of the invention, as will be apparent to those skilled in the art. Other embodiments derived from this specification will also be obvious to those skilled in the art. This application specification and embodiments are merely exemplary.

[0052] The terms “include,” “including,” “have,” “contain,” etc., used in this article are all open-ended terms, meaning that they include but are not limited to.

[0053] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0054] like Figure 1 - Figure 2 As shown, this embodiment provides a table recognition method based on a GNN and OCR system, including: acquiring a table image to be recognized; extracting image visual features and text semantic features from the table image to be recognized, fusing the extracted image visual features and text semantic features to obtain a fused feature map; inputting the fused feature map into a segmentation model to predict the physical layout of cells, wherein the segmentation model is built based on the CellPose model; inferring the row and column relationships between cells based on the physical layout of cells, combined with a graph neural network; performing post-processing based on the inference results of the graph neural network, specifically including: assigning all cells to the correct row and column indices, and assigning semantic labels to each cell; and outputting a table recognition result in a preset format after post-processing.

[0055] Traditional table recognition methods rely on OCR accuracy and wireframe integrity, resulting in low recognition rates for complex tables. Furthermore, their complex model structures and high training costs limit their practical application in industrial scenarios. To address these issues, this embodiment provides a table recognition method based on a GNN and OCR system. The method includes: extracting visual features and textual semantic features from the input image using a Transformer-based visual model and an OCR system, respectively; performing early fusion through a cross-attention mechanism to obtain a fused feature map containing contextual information; further, inputting the fused feature map into a segmentation network based on the CellPose model to segment the instance mask of each cell and generate a cell physical layout proposal; constructing a graph structure based on the cell physical layout; using a graph neural network to infer row and column relationships between cells; finally, based on the GNN inference results, allocating row and column indices using a disjoint-set data structure algorithm; identifying merged cells using geometric rules; and assigning semantic labels using a named entity recognition model, ultimately outputting a structured JSON-formatted recognition result. This embodiment effectively solves the challenges of complex table structure recognition, wireless table processing, and semantic understanding, significantly improving the accuracy and automation of table recognition.

[0056] The specific implementation process of this embodiment includes:

[0057] (1) Image input and preprocessing:

[0058] The process involves acquiring the image of the table to be recognized. Input images can include scanned paper documents, photographs, or electronic files. The input image is then enhanced by performing operations such as noise reduction, contrast enhancement, and image correction, depending on the image quality, to improve the accuracy of subsequent recognition.

[0059] The image needs to be scaled to a fixed size for model processing to ensure consistency in processing.

[0060] (2) Extraction and fusion of multimodal features:

[0061] The preprocessed image is used to extract deep features using a visual backbone network based on the Transformer architecture (Swin Transformer), and a high-resolution visual feature map is output.

[0062] Text feature extraction based on PaddleOCR system:

[0063] The text detection model (DBNet++) is used to locate the positions of all text lines in the image and obtain their bounding box coordinates. Then, the text recognition model is used to identify each detected text region to obtain the specific text content and its confidence score.

[0064] Each detected text unit (including its content, coordinates, and confidence score) is defined as a text token, and all text tokens are grouped into a list. The string content of each text token is input into a pre-trained text encoder (BERT) to convert it into a fixed-length text feature vector.

[0065] Feature fusion: The bounding box coordinates of each text token are mapped onto a visual feature map to find its corresponding visual region. Based on a cross-attention mechanism, the text feature vector of the text token is fused with the visual feature vector of its corresponding visual region, simultaneously encoding visual appearance and textual semantic information, providing a foundation for subsequent cell segmentation.

[0066] By using multimodal feature fusion technology, the model incorporates textual semantic information while perceiving images, thus effectively dealing with complex scenarios such as wireless tables and distorted tables. Its recognition accuracy and robustness are far superior to traditional methods that rely on line detection.

[0067] (3) Cell instance splitting:

[0068] The CellPose architecture is adopted. The advantage of the CellPose model is that it can handle adjacent and connected objects. This model can handle closely adjacent cells and avoid complex line prediction and post-processing.

[0069] The model processing includes:

[0070] The decoder of CellPose is used, while a lightweight ResNet18 model is used as the encoder of CellPose.

[0071] The CellPose training process is as follows: the training data is labeled, which includes the initial image and the labeled image. Each pixel in the labeled image needs to be marked as belonging to a specific cell. The training data is then input into the model, and the model is trained using a combined loss function of segmentation loss and flow field regression loss to guide the model to correctly separate the adhered instances.

[0072] The decoder outputs two key quantities: a probability map and a flow map. The probability map represents the probability that each pixel belongs to any cell. The flow map predicts a vector pointing to the center of its cell for each foreground pixel.

[0073] Using the predicted flow field map, pixels are grouped to their respective cell centers, forming an instance segmentation mask for each cell. For each segmented cell instance mask, its minimum bounding rectangle is calculated to obtain the cell coordinates.

[0074] Finally, based on the area, aspect ratio, and prediction probability of the bounding box, invalid proposals that are too small or have too low confidence are filtered out, and the cell physical layout proposals including coordinates and segmentation masks are output.

[0075] (4) Graph construction and relational reasoning:

[0076] To convert the physical set of cells into a graph data structure, first, define the nodes and edges:

[0077] Each cell in a proposal becomes a node in the graph, and each node corresponds to a feature vector for representation.

[0078] Visual feature vector: On the visual feature map, a fixed-length feature vector is extracted using the RoI Align operation based on the bounding box coordinates of the cell. This vector encodes the visual appearance information of the cell region.

[0079] Geometric feature vector: Encodes the geometric information of a cell into a vector and then normalizes it.

[0080] The visual feature vector and the geometric feature vector are concatenated to obtain the final total feature vector.

[0081] The k-Nearest Neighbors (k-NN) algorithm is used to construct edges based on the coordinates of the cell's center point.

[0082] Calculate the coordinates of the center point of all cells to form a point set. For each node, find the k nearest neighbors in Euclidean distance and create undirected edges in the graph to connect the node with these k neighboring nodes.

[0083] Graph Neural Network Inference:

[0084] Based on the GNN model, the relationships between nodes (cells) in the graph are inferred, and the relationship type of each edge in the graph is predicted. The specific process is as follows:

[0085] GNN updates node features through a multi-layered "message passing" mechanism, with each layer performing the following operations:

[0086] For each edge in the graph, its source node generates a message, which is the node's current feature vector. Each node receives messages from all its neighboring nodes, aggregates the messages to form a single summary message. The node concatenates its current features with the aggregated summary message, and the concatenated message is input into a fully connected neural network (MLP) to output a new feature representation.

[0087] After several layers of message passing, we obtain the final node features rich in context, and then further predict the category of each edge:

[0088] For each edge, the final feature vectors of its two endpoints are input into a multilayer perceptron (MLP) model for three-class classification to obtain the probability that the edge belongs to each category (irrelevant, same row, and same column).

[0089] (5) Post-processing:

[0090] Collect all edges predicted as peers and all edges predicted as co-pairs, and construct a relation matrix.

[0091] Use the Union-Find algorithm to allocate row and column indexes:

[0092] Initialize a disjoint-set data structure, where each node forms its own independent set. Traverse all edges in the same row and merge the sets containing the two nodes connected by the edge. Each connected component in the disjoint-set data structure represents a row in the table. All cell nodes within the same component belong to the same row.

[0093] Calculate the average Y-coordinate of all nodes within each connected component, and sort the connected components from top to bottom based on the average Y-coordinate. The sorted structure is the row index of all cells in that row.

[0094] Similarly, using the edges of the same column relationship, all columns are found through the disjoint set and the column indexes are assigned in order from left to right.

[0095] Inferring merged cells:

[0096] Calculate the average height of all cells in each row and the average width of all cells in each column. Iterate through each cell, comparing its height with the average height of its row and its width with the average width of its column.

[0097] Semantic processing: The text content recognized by OCR is allocated and filled into the corresponding cells according to the overlap relationship between its bounding box coordinates and cell coordinates.

[0098] Train a named entity recognition model (DistilBERT model) to automatically determine the semantic category of the text content in a cell:

[0099] The required training data includes plain text strings and corresponding semantic labels. The training data is input into the constructed initial model, and the training is carried out with the goal of minimizing the loss between the initial prediction results after classification and the semantic labels corresponding to the plain text strings, so as to obtain a trained named entity recognition model.

[0100] A lightweight named entity recognition model is used to classify and predict the text content of each cell to obtain semantic tags, including headers, data, dates, and amounts, thereby understanding the purpose of each cell.

[0101] (6) Serialization and output: Integrate all information, assemble the physical coordinates of the cells, logical row and column indices, cross-row and cross-column information, text content, semantic tags and the overall information of the table, and serialize them into a structured JSON data.

[0102] This JSON can be directly stored in a database or provided to downstream applications, achieving a complete transformation from images to structured data.

[0103] In summary, this embodiment accurately segments cell entities through parallel visual and text feature extraction and deep fusion, then infers their logical structure and merging relationships using graph neural networks, and finally outputs directly usable structured data using semantic understanding. The entire process is highly automated and robust to complex and special tables.

[0104] This embodiment also provides a table recognition system based on GNN and OCR systems, including:

[0105] The data acquisition module is used to acquire the image of the table to be recognized;

[0106] The multimodal feature processing module is used to extract the image visual features and text semantic features of the table image to be identified, and to fuse the extracted image visual features and text semantic features to obtain a fused feature map;

[0107] A cell detection module is used to input the fused feature map into the segmentation model to predict the physical layout of cells, wherein the segmentation model is built based on the CellPose model;

[0108] The row and column relationship reasoning module is used to infer the row and column relationships between cells based on the physical layout of the cells and in conjunction with a graph neural network.

[0109] The post-processing module is used to perform post-processing based on the inference results of the graph neural network. Specifically, it includes: assigning all cells to the correct row and column indices and assigning semantic labels to each cell.

[0110] The output module is used to output the table recognition results in a preset format.

[0111] The above description is merely a preferred embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A table recognition method based on a GNN and an OCR system, characterized in that, The method comprises the following steps: obtaining a table image to be recognized; extracting image visual features and text semantic features of the table image to be recognized, fusing the extracted image visual features and text semantic features to obtain a fused feature map; inputting the fused feature map into a segmentation model to predict the physical layout of a cell, wherein the segmentation model is constructed based on a CellPose model; combining graph neural network to infer the row-column relationship between cells based on the physical layout of the cell; post-processing based on the inference result of the graph neural network, specifically including: assigning all cells to correct row and column indexes and assigning semantic labels to each cell; outputting a table recognition result in a preset format after the post-processing is completed.

2. The method of claim 1, wherein, The process of obtaining the image visual features specifically includes: extracting image visual features of the table image to be recognized by using a visual model based on Transformer to obtain a visual feature map.

3. The method of claim 2, wherein, The process of obtaining the text semantic features specifically includes: performing text detection and text recognition on the table image to be recognized based on an OCR system to obtain a plurality of text tokens, wherein the text tokens include text content and a bounding box coordinate; converting the text content of each text token into a text feature vector; generating a text token list to complete the extraction of text semantic features.

4. The method of claim 3, wherein, The process of obtaining the fused feature map specifically includes: mapping the bounding box of each text token to the visual feature map, and splicing the corresponding text feature vector and the visual feature vector to obtain the fused feature map.

5. The method of claim 1, wherein, The process of inputting the fused feature map into the segmentation model to predict the physical layout of the cell specifically includes: performing segmentation on the fused feature map based on the CellPose model to obtain segmented cell instance masks; calculating the minimum bounding rectangle of each cell instance mask to obtain the coordinates of the cell, and generating a preliminary cell proposal based on the cell instance mask and the coordinates.

6. The method of claim 1, wherein, The process of combining the graph neural network to infer the row-column relationship between cells specifically includes: constructing a graph structure data, taking each cell as a node, and each node corresponding to a feature vector, wherein the feature vector includes visual features and geometric features; constructing an edge by using a K-nearest neighbor algorithm, and connecting each cell to the corresponding k nearest neighbors in space based on the coordinates of the cell center point; inputting the constructed graph structure data into the graph neural network to predict the row-column relationship between the two cells connected by each edge, wherein the row-column relationship includes irrelevant, same row and same column.

7. The method of claim 6, wherein, The process of post-processing based on the inference result of the graph neural network specifically includes: obtaining a same-row relationship edge set, finding all rows according to the union-find set algorithm, sorting the rows from top to bottom according to the average Y coordinate of the rows, and assigning row indexes; obtaining a same-column relationship edge set, finding all columns according to the union-find set algorithm, sorting the columns from left to right according to the average X coordinate of the columns, and assigning column indexes; calculating the average height of all cells in each row and the average width of all cells in each column; for each cell, if the physical size of the cell is greater than the average size of the corresponding row and column, the current cell is a merged cell; According to the coordinate overlap, the text semantic features are matched into the corresponding cells; Based on the named entity recognition model, the text content of each cell is classified and predicted to obtain the semantic label corresponding to each cell; The physical layout, row and column structure analysis and semantic label corresponding to each table are serialized to output the structured table recognition result. 8.A table recognition system based on GNN and OCR system, characterized in that, It comprises: a data acquisition module configured to acquire a table image to be recognized; a multi-modal feature processing module configured to extract image visual features and text semantic features of the table image to be recognized, and fuse the extracted image visual features and text semantic features to obtain a fused feature map; a cell detection module configured to input the fused feature map into a segmentation model to predict the physical layout of the cells, wherein the segmentation model is constructed based on a CellPose model; a relationship reasoning module configured to infer the row and column relationship between the cells according to the physical layout of the cells and in combination with a graph neural network; a post-processing module configured to perform post-processing according to the inference result of the graph neural network, specifically including: assigning all cells to correct row indexes and column indexes, and assigning a semantic label to each cell; an output module configured to output a table recognition result in a preset format.

Images