A table structure recognition method based on vertex detection

Abstract

The present application relates to a kind of table structure identification method based on vertex detection, utilize the table line and table vertex information of table picture, for given picture, using vertex detection model to detect table vertex, using edge detection method to detect table line information, obtain table cell information comprehensively, finally analysis obtains table structure.The present application has the advantages that: only a simple key point detection model is used for the detection of vertex, so the method requires less computing resources and occupies less, and has better real-time performance;It can solve the difficulties of variable table style and complex shooting scene in practical application, and for the new scene without training, it does not need to re-label the table of the scene with a large amount of manual work, so that a high accuracy can be maintained, and for the simple table picture with a large proportion, the effect is more stable and accurate, and it is not easy to miss.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a table structure recognition method based on vertex detection. Background Technology

[0002] With the rapid development of information technology and the continuous growth of data, table structure recognition has become increasingly important. In modern society, people face a large amount of data contained in tables, which often contain important and rich information.

[0003] With the widespread adoption of the internet, a vast amount of documents, reports, and publicly available datasets exist online. These resources frequently use tables to organize complex data. Along with the accelerating pace of digital transformation, many traditional paper documents are being digitized for storage and management. Large amounts of historical data, as well as real-time generated data, exist in tabular form in enterprise and organizational databases. Through table structure recognition, we can transform this massive amount of tabular data into a structured format, making it easy to query, analyze, and utilize. This transformation not only improves data processing efficiency but also provides a solid foundation for business decision-making and strategic planning. Furthermore, table structure recognition is crucial for the development of natural language processing and artificial intelligence. Many natural language processing tasks require extracting information from tables, such as question-answering systems and knowledge graph construction. By accurately recognizing table structures, we can better understand the semantic information within tables and combine it with textual data to achieve higher levels of semantic understanding and reasoning.

[0004] In conclusion, with the rapid development of information technology and data, table structure recognition plays a crucial role in data analysis, knowledge management, and artificial intelligence applications. It not only helps us better utilize massive amounts of tabular data but also promotes cross-disciplinary research and innovation. Therefore, strengthening research and application of table structure recognition technology is of great significance for further advancing the information age.

[0005] In recent years, with the rapid development of deep learning technology, table structure recognition has also made significant progress. Deep learning-based table structure recognition methods can be broadly categorized into methods based on cell detection and relationship classification, methods based on row and column segmentation and cell merging, and methods based on image-to-table structure sequence generation. Representative methods based on cell detection and relationship classification include Cascade R-CNN, whose main idea is to use detection or segmentation methods to obtain the cells containing the text detection boxes in the table, and then use graph networks or other classification models to predict the relationships between cells. A representative method based on row and column segmentation and cell merging is DeepDeSRT, which divides the table into basic units by detecting row and column separators, and then uses a merging module to combine these basic units. Methods based on image-to-table structure sequence generation, such as EDD, directly predict the text sequence of the HTML representation of the table structure using LSTM or transformers, but currently, this method's performance is not stable enough.

[0006] The above method works well only for simple tables synthesized from PDFs. In practical applications, there are difficulties such as varied table styles and complex shooting scenarios. The above method cannot automatically adapt to new scenarios, and a lot of manual annotation is required for tables in different scenarios to make the model adapt to the changing scenarios. Summary of the Invention

[0007] The purpose of this invention is to address the shortcomings of the prior art by providing a table structure recognition method based on vertex detection. This method utilizes the table lines and vertex information of a table image and employs a combination of vertex detection and edge extraction to recognize the table structure. This effectively solves the difficulties encountered in applications, such as the variety of table styles and the complexity of shooting scenarios, and achieves better, more stable, and accurate results.

[0008] The objective of this invention is achieved through the following technical solutions:

[0009] Step S1: Obtain the table image to be identified. Input the table image to be identified into the key point detection model. The model will generate a heatmap of the table vertices based on the table image and obtain the vertex coordinates of the table cells through the heatmap.

[0010] Step S2: Use erosion, dilation and edge detection methods to process the original table image, and then perform binarization to obtain the approximate outline of the table lines.

[0011] Step S3: For the predicted table vertices, combine the outline of the table lines and use a breadth-first search method to connect vertices belonging to the same cell and separate cells belonging to different tables.

[0012] Step S4: Use text detection and recognition methods to detect and recognize the text in each cell.

[0013] Step S5: Based on the coordinates of the four sides of each cell in the table, obtain the row and column relationships between cells. Based on the row and column relationships of the cells in the entire table, obtain the starting row and column number of the table to which the cell belongs.

[0014] Step S6: Reconstruct the structure of all tables in the entire image based on the starting row and column number of each table cell and the text detected and recognized.

[0015] A training phase is required before inference deployment:

[0016] Data collection steps: Collect publicly available datasets for table structure recognition and filter data containing table vertex annotations as training data.

[0017] Model training steps: Train the keypoint detection model using the collected public dataset. Optimize the model using the stochastic gradient descent algorithm so that the model can accurately predict the coordinates of the four vertices of the table.

[0018] The table structure recognition method based on vertex detection provided by this invention may also have the following technical features: the vertex detection model used is based on the CenterNet key point detection model, and the two-dimensional normal distribution heatmap corresponding to the coordinate positions of all vertices of each table is used as a preset pre-trained label.

[0019] The table structure recognition method based on vertex detection provided by this invention may also have the following technical features: the method for obtaining the table line contour is to perform Gaussian blur on the table image and then use the Laplacian operator for edge extraction.

[0020] The table structure recognition method based on vertex detection provided by this invention also has the following technical feature: the mean squared error loss calculation formula used for calculating the loss is as follows:

[0021]

[0022] The advantages of this invention are:

[0023] 1) It can effectively solve the difficulties faced in practical applications, such as the variety of table styles and the complexity of shooting scenarios.

[0024] 2) Since the method uses a deep learning model for table vertex detection that is not sensitive to the scene, it can maintain high accuracy without having to manually annotate a lot of tables in a new scene that has not been trained.

[0025] 3) The method of combining vertex detection and edge extraction, which uses the table lines and vertex information of the table image, is less prone to errors compared to the pure deep learning method. Moreover, it is more stable and accurate for simple table images with a large proportion of the image, making it more suitable for commercial use.

[0026] 4) Only a simple key point detection model is used for vertex detection, so the method has low requirements and consumption of computing resources and good real-time performance. Attached Figure Description

[0027] Figure 1 This is a flowchart of the present invention. Detailed Implementation

[0028] The following examples further illustrate the features and other related characteristics of the present invention in detail, to facilitate understanding by those skilled in the art:

[0029] Example: Figure 1 As shown in this embodiment, the table structure recognition method based on vertex detection includes the following steps in the inference stage of table structure recognition:

[0030] Step S1: For the table image to be identified, input the table image into the key point detection model. The model will generate a heatmap H of the table vertices based on the table image, and obtain several points based on the peak values ​​of the heatmap. Then, the values ​​of these points are filtered using a threshold α to obtain the vertex coordinates P of the table cells in the table. p ,in

[0031] Step S2: Use Gaussian to blur the original image, then use the Laplacian edge extraction method to obtain the approximate outline of the table lines, and then use binarization to obtain a continuous edge map M, where the value is 1 for points with edges and 0 otherwise.

[0032] Step S3: For the predicted table vertex P p And the outline edge map M of the table lines, adaptively draw a circle with radius r on M. p The value of the circle is -P, so that each vertex P is... p Solid circles with different values ​​are drawn at the center of each image (because the image color type is uint8, which is actually equivalent to 255-p). This will ultimately assign a value of 0 to all blank areas in the image, a value of 1 to the table lines, and a value of 255-p around the vertices, where r... p Just enough to satisfy the condition that vertex P can be p The edge graph at point P is completely covered, ensuring that there are no more connections between any vertices. Then, a breadth-first traversal method is used to traverse each vertex P. pStarting from point M, search for other reachable vertices along the path with values ​​of 1 or -p. Stop searching when a value of 0 is encountered, and stop searching when a value of -p is encountered. j When, this indicates point P p and There are edges between them, that is, lines belonging to the same cell. Finally, a graph showing the relationships between all vertices of the entire table and their adjacent vertices is obtained. From the graph, the coordinates B of the four vertices of each cell can be analyzed. c Each connected component in the diagram represents a complete table in the table image.

[0033] Step S4: Using text detection and recognition methods, detect and recognize the text in each cell to obtain T. c .

[0034] Step S5: Based on the coordinates of the four sides of each cell in the table and the adjacency diagram, obtain the row and column relationships between cells. and Based on the row and column relationships of the cells in the entire table, and for B c The horizontal and vertical edges in the table are sorted by coordinates, and the starting row and column number (Row_start) of the cell is obtained by sorting the rows and columns according to the row order. c Row_end c Col_start c Col_end c .

[0035] Step S6: Reconstruct the structure of all tables in the entire image based on the starting row and column number of each table cell and the text detected and recognized.

[0036] A training phase is required before inference deployment:

[0037] Data collection steps: Collect publicly available datasets for table structure recognition and filter those containing table vertex annotations for training. The WTW dataset is typically the primary dataset used.

[0038] Model training steps: First, a keypoint detection model based on CenterNet is built as the model used for vertex detection. The model uses ResNet as the backbone network, then uses FPN to enhance features, and finally inputs the resulting data into an MLP to obtain a heatmap. The heatmaps corresponding to the coordinates of all vertices in each table are used as pre-trained labels. The model is trained using mean squared error loss. Optimization was performed using the stochastic gradient descent algorithm.

[0039] This embodiment utilizes the table vertex detection result P pBy connecting the edge detection results M of the table image to the four vertices of each cell, we obtain the cell coordinates B. c And obtain the text T within the cell through a text detection and recognition model. c Based on the coordinates B of all cells c Obtain row and column adjacency pairs and Then obtain the starting row and column number of the cell, Row_start. c Row_end c Col_start c Col_end c Finally, the above results were integrated and the table structure was reconstructed.

[0040] In this embodiment, the pre-trained text detection and recognition model achieves better results on table images. The table vertex detection model, after training on the required dataset, achieves better results.

[0041] Although the above embodiments have described the concept and embodiments of the present invention in detail, those skilled in the art will recognize that various improvements and modifications can still be made to the present invention without departing from the scope of the claims, and therefore will not be elaborated here.

Claims

1. A table structure recognition method based on vertex detection, characterized in that: Includes the following steps: S1. Obtain the table image to be identified, input the table image to be identified into the key point detection model, the key point detection model generates a heat map of the table vertices based on the table image, and obtains the vertex coordinates of the table cells in the table image through the heat map; S2. Using erosion dilation and edge detection methods, process the original table image to be identified, and then perform binarization to obtain the outline of the table lines; S3. Combining the vertex coordinates predicted by S1 and the outline of the table lines obtained by S2, a breadth-first search method is used to connect vertices belonging to the same cell and separate cells belonging to different tables. S4. Use text detection and recognition methods to detect and recognize the text in each cell; S5. Based on the coordinates of the four sides of each cell in the table, obtain the relationship between rows and columns of the cells; Based on the row and column relationships of the cells in the entire table, obtain the starting row and column number of the table to which the cell belongs; S6. Based on the starting row and column number of each table cell and the text detected and recognized, reconstruct the structure of all tables in the entire image to be recognized; For the predicted table vertices Outline edge diagram of table lines ,exist Adaptively draw a radius of The value is The dots in the image are of type uint8, which is actually equivalent to 255- ,in Just enough to satisfy the condition that the vertex The edge graph is completely covered at each vertex, ensuring that there are no connections between any vertices. Then, a breadth-first traversal method is used to traverse from each vertex. Departure, at The upper edge value is 1 or The search continues until a value of 0 is encountered, at which point the search stops. The search continues until a value of 1 is encountered. At that time, explain the point. and There are edges between them, that is, lines belonging to the same cell; finally, a graph showing the relationship between all vertices of the entire table and their adjacent vertices is obtained; from the graph, the coordinates of the four vertices of each cell are analyzed. In the diagram, each connected component represents a complete table in the table image.

2. The table structure recognition method based on vertex detection according to claim 1, characterized in that: The keypoint detection model iterates through publicly available table structure recognition datasets that contain table vertex annotation data, and optimizes the model using a stochastic gradient descent algorithm, enabling the keypoint detection model to predict the coordinates of the four vertices of the table.

3. The table structure recognition method based on vertex detection according to claim 2, characterized in that: The keypoint detection model is a CenterNet-based keypoint detection model, which uses ResNet as the backbone network, then uses FPN to enhance features, and finally upsamples and inputs into MLP to obtain a heatmap.

4. The table structure recognition method based on vertex detection according to claim 2, characterized in that: The key point detection model uses a two-dimensional normal distribution heatmap corresponding to the coordinate positions of all vertices in each table as a preset pre-training label.

5. The table structure recognition method based on vertex detection according to claim 1, characterized in that: In step S2, the original image is blurred using Gaussian, and then the outline of the table lines is obtained using the Laplacian edge extraction method. Then, the image is binarized to obtain a continuous edge map, where the value is 1 for points with edges and 0 otherwise.

6. The table structure recognition method based on vertex detection according to claim 1, characterized in that: Based on the coordinates of the four sides of each cell in the table and the adjacency diagram, the relationships between rows and columns of the cells are obtained. and Based on the row and column relationships of the cells in the entire table, and on... By arranging the horizontal and vertical edges of the table according to their coordinates and sorting them by sequence number, the starting row and column number of the cell can be obtained. , , , .

Images