Scientific literature-oriented complex table structure identification method

Abstract

The invention discloses a scientific literature-oriented complex table structure identification method. The method comprises the following steps of (1) firstly, positioning and cutting table areas in scientific literatures; (2) inputting the cut table picture into an LGPMA network model to perform feature extraction of a text region; (3) the local branch obtains information of the text region through a texture perceptron; (4) the global branch is used for obtaining space information such as distance between text blocks in the global; (5) obtaining initial text block position information by using a mask re-scoring strategy and fully utilizing local and global feature information; and (6) providing a method for dividing vertical parting lines to solve the adhesion condition between the cells. And (7) grouping according to the text blocks to obtain the positions of the text blocks in the table, using a post algorithm to solve the problem of merging the cells, and storing the table in an html format. And the adjacent position relationship between the cells is effectively reserved by using the text region detection based on the network model, so that the recovered table structure is more accurate.

Description

Technical Field

[0001] The present invention relates to the field of table structure recognition, and in particular to a complex table structure recognition method for scientific literature. Background Art

[0002] In many application scenarios, tables are an efficient tool that can display information intuitively. Compared with ordinary text, tables express the relationship between data more accurately through a structured format and are concise and clear. Its structured characteristics give tables significant advantages in information transmission, data organization and aggregation. Especially in complex data processing, tables can clearly display information from different sources and provide basic support for subsequent operations (such as sorting, filtering and data joining). Therefore, tables are not only an important tool for information collection and organization, but also can transform information into knowledge through data analysis to provide support for decision-making. Precisely because of its convenience and practicality, tables have been an important object in academic research and industrial applications over the past two decades.

[0003] The main goal of table structure recognition is to automatically extract the structured information of tables from various documents, which is particularly important in scenarios that require accurate data extraction, such as scientific literature, financial reports, and contracts. Traditional table recognition methods usually rely on rules and mathematical algorithms, such as boundary-based segmentation algorithms, heuristic rules, and manually designed feature extraction methods. These methods perform well when dealing with simple regular tables, but when faced with complex layouts, borderless tables, or tables with special structures such as merged cells, traditional methods often seem powerless. Due to the lack of in-depth understanding of global and local features, traditional methods have low recognition accuracy when dealing with complex tables that span pages, merge multiple columns or rows, and are difficult to meet the needs of practical applications.

[0004] With the rapid development of deep learning technology, table recognition methods based on convolutional neural networks (CNN), graph neural networks (GNN) and other neural network architectures have gradually become mainstream. Typical deep learning methods such as the Mask-RCNN-based table recognition model and TableNet handle table structure recognition tasks in an end-to-end manner, significantly improving the accuracy and efficiency of table structure extraction. These models can automatically detect text areas in tables and accurately identify row and column information, and can even handle complex borderless tables, merged cells, and multi-page tables. However, despite the breakthroughs in performance of deep learning methods, they also face some practical challenges.

[0005] First, the challenges brought by the diversity and complexity of table layouts are particularly prominent. Tables in documents come in various forms, some may not have clear borders, and some may contain complex merged cell structures, which makes it difficult for deep learning models to extract accurate row and column information. In addition, noise in table documents, such as blur, tilt or other distortion from scanned files, further increases the difficulty of recognition, resulting in the inability to accurately extract features in certain areas.

[0006] Secondly, the limited number of annotated datasets limits the generalization ability of the model. Although public datasets such as ICDAR and PubTabNet provide important training resources for table recognition research, these datasets are often not comprehensive enough in practical application scenarios with complex table structures, and it is difficult to cover all table types and layout variations. The model may show inconsistent performance when facing documents from different sources, especially when the table formats in the documents are diverse. This problem is more obvious.

[0007] In response to these challenges, current research is gradually exploring hybrid strategies that combine deep learning with traditional methods. Deep learning models can be used to extract local and global features of complex tables, while traditional heuristic algorithms can be used as post-processing steps to solve problems such as table segmentation and cell merging. Through this combination, the feature extraction capabilities of deep learning and the flexibility of heuristic algorithms can be taken into account when processing complex tables, effectively improving recognition accuracy.

[0008] In general, the field of table structure recognition is developing rapidly. With the combination of deep learning and traditional algorithms, table structure recognition technology will be more widely used in the future and will show stronger capabilities in processing more complex document tables. Summary of the invention

[0009] Purpose of the invention: In view of the shortcomings of existing table structure recognition methods in dealing with complex table layouts, borderless tables and cell adhesion problems, the present invention proposes a table structure recognition method based on a combination of deep learning and heuristic algorithms. By combining local and global feature extraction, the recognition accuracy of the table structure is improved. And by using vertical dividing lines to split text blocks, the cell adhesion problem is solved, and finally the accurate recognition and restoration of complex tables is achieved.

[0010] Technical solution: In order to achieve the above invention objectives, the technical solution of the present invention is as follows:

[0011] (1) Table area positioning and cropping: First, the table area in the document is automatically positioned and cropped through the image processing algorithm to ensure that the processed image only contains the table part, reducing the interference of non-table information on recognition.

[0012] (2) Deep learning feature extraction: The cropped table image is input into the deep learning model based on the LGPMA network. The model is based on two branches: local feature extraction and global feature extraction, which process the local information and spatial position information of the text box respectively, so as to obtain the accurate text area and the corresponding row and column relationship.

[0013] (3) Fusion of local and global features: By combining the texture information extracted by the local branch with the spatial distance information of the global branch, the exact position of each text block in the table is further determined.

[0014] (4) Vertical dividing line division: To solve the problem of cell adhesion, a vertical dividing line method is used to separate adjacent text blocks. By detecting the boundary of each text block, it is determined whether a dividing line needs to be inserted, thereby ensuring the independence of the cell content.

[0015] (5) Text block grouping and merged cell processing: According to the extracted text block positions, the text blocks are grouped to ensure that the content of each cell is correctly allocated. For merged cells, the post-processing algorithm is applied to ensure that the structure of the merged cells is correctly restored.

[0016] (6) Table structure restoration and storage: The table structure is finally stored in HTML format, and the row and column layout and cell merging relationship of the table are completely restored to achieve visual presentation of the table and subsequent data analysis.

[0017] Preferably, in step (1), positioning and cutting the table area specifically includes the following steps:

[0018] (1) First, convert the document PDF format into jpg image format, and perform preliminary positioning and detection based on the TableBank table area detection model.

[0019] (2) Based on the heuristic scanning boundary line algorithm, the table area boundary is finely located. This step prevents the data in the table area from being missed.

[0020] Preferably, in step (4), the vertical dividing line is divided to solve the problem of text contents between cells sticking to each other, which specifically includes the following steps:

[0021] (a) First, through the text detection in the previous step, the coordinates of the text box area of ​​each cell in the table are obtained, and the right boundary line is uniformly selected as the segmentation line.

[0022] (b) The average value of the distance between cells in the table is used as a threshold to determine whether the contents of two adjacent cells belong to one cell or two cells, and the correct table content is extracted in conjunction with the table extraction algorithm.

[0023] Preferably, in step (5), the text blocks extracted by the table extraction algorithm need to be grouped and processed by a post-processing algorithm. Specifically, the following steps are included:

[0024] (a) The extracted text blocks are grouped according to the row and column information, that is, the extracted table data is stored in a two-dimensional matrix format.

[0025] (b) The post-processing algorithm here mainly determines whether there are merged cells by detecting whether the number of row and column cells is the same. For example, if the number of column cells in the first row is less than that in the second row and the number in the second row is a multiple of the first row, the first row is considered to be a merged cell.

[0026] Preferably, in step (6), in order to restore the final table structure, a structured data structure needs to be used for storage, and HTML naturally supports the storage of such table structure. Specifically, the following steps are included:

[0027] (a) First check whether there are merged cells in the extracted table text block. If necessary, add rowspan or colspan in the corresponding tags.

[0028] (b) Other cells, such as column headers and row headers, are stored according to the corresponding tree structure.

[0029] (c) So far, all cell information and table structure information are stored in the HTML file in a tree structure. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is a specific flow chart of the present invention;

[0031] Figure 2 It is to extract the table flow chart from the page through the algorithm; DETAILED DESCRIPTION

[0032] The technical solution of the present invention is further described below in conjunction with the accompanying drawings:

[0033] Step (1) Get the page text element. The text in the PDF file is not stored in a structured manner like a word document. It is usually saved in the form of drawn characters, coordinate positions, fonts, and sizes. Figure 2 As shown, the specific steps are as follows:

[0034] Step (1a) parses the content stream of the PDF. This step extracts relevant information of the text, such as the Unicode encoding of the characters, coordinate position, font size, and height and width, and parses them into objects.

[0035] Step (1b): Check whether the text is repeated or overlapped, and determine whether the text is a valid character.

[0036] Step (1c): The text in the PDF document does not have clear line and paragraph definitions, so it is necessary to analyze the vertical coordinates (Y coordinates) of the text to determine whether it belongs to the same line, and analyze the horizontal spacing between characters (X coordinates) to determine whether spaces or line breaks need to be added:

[0037] o Two characters are considered to be in the same row if their Y coordinates are close.

[0038] oIf the distance between the X coordinates is larger, a space is inserted to indicate that these are two words. :

[0039] Step (2) requires merging the text blocks extracted in step 1, such as Figure 2 As shown:

[0040] Step (2a): First, loop through each text element and check whether the font size of the current character is the same as the previous character. If not, reset and recalculate the character spacing:

[0041] avg=(preAvg+(chr.getWidth() / chr.getText().length())) / 2

[0042] This formula adds the previous average width to the current character width and divides it by 2 to form the new average width. This method ensures a smooth transition of character widths, especially when the widths of consecutive characters do not vary much.

[0043] Step (2b): If a reset is required, the average character width must be recalculated based on the current character, that is, the current character width divided by the current character length.

[0044] avg=chr.getWidth() / chr.getText().length()

[0045] Step (3) determines whether there is a vertical dividing line between the current character and the previous character. If the two characters are on different sides of the vertical dividing line, they will not be merged.

[0046] Step (4) Calculation of character spacing:

[0047] Step (4a): First estimate the expected distance between text boxes based on the space width of the characters. If the space width is not available or is zero, it is set to the maximum value. If the space width has not been calculated before, use half of the current space width as the tolerance:

[0048]

[0049] Step (4b): Estimate the reasonable distance between two text blocks based on the average width of the characters. By comparing the width of the current character with the average width of the previous characters, the average character width is obtained, and multiplied by a tolerance value to calculate the reasonable distance between characters:

[0050]

[0051] deltaCharWidth=avgCharWidth×AVG_CHAR_TOLERA

[0052] Step (4c): Compare the tolerance of the space width and the character width, and select the smaller value as the distance between the expected text blocks. Finally, determine the starting position of the next text block by the right boundary position of the previous text block and the minimum spacing:

[0053] expectedStartOfNextWordX

[0054] =endOfLastTextX+min(deltaSpace, daltaCharWidhth) Step (5) Determine whether they are in the same line and merge text blocks:

[0055] Step (5a): First, compare the vertical coordinates of the current character and the previous character to determine whether they are in the same line. If the vertical coordinates do not overlap, they are considered to be in different lines and the relevant parameters such as the maximum line height and the right boundary of the text block are reset.

[0056] Step (5b): If the characters are on the same line, check whether there is a vertical separator between the current character and the previous character. If there is no vertical separator, the character will be merged into the existing text block.

[0057] Step (5c): If the current character and the previous character are not in the same line or are separated by a vertical line, a new text block will be created and the current character will be added to the new block.

[0058] Step (6) Group text blocks by directionality:

[0059] Step (6a): Check the character directionality (LTR or RTL) of the text block to determine whether the text block is from left to right or from right to left.

[0060] Step (6b): Group the text blocks according to their directionality to ensure that there is no direction confusion when processing bidirectional text.

[0061] Step (6c): Finally, the text blocks grouped by directionality are returned to ensure that texts in multiple languages ​​and different directions are processed correctly.

[0062] The present invention effectively reduces the sticking of table cells in the post-processing of text blocks by adding vertical dividing line information to the text, ensuring the accuracy and consistency of the extracted table data, and simplifies the processing of text blocks by calculating the character distance and the judgment of text grouping in the algorithm, which not only ensures that the characters are distributed by rows, but also can reasonably group multi-directional texts.

Claims

1. A complex table structure recognition method for scientific literature, characterized in that: The following steps are involved: (1) Locate and crop table areas in scientific literature; (2) Input the cropped table image into the Mask-RCNN network model for text feature extraction; (3) The local branch extracts the texture information of the text area; (4) The global branch obtains the spatial distance information between text blocks; (5) Determine the preliminary position information of the text block by combining local and global features through the mask re-scoring strategy; (6) Use the vertical dividing line method to solve the cell adhesion problem; (7) Determine the position of text blocks in the table by grouping, apply the post-placement algorithm to solve the problem of merging cells, and store the table in HTML format.

2. A complex table structure recognition method for scientific literature according to claim 1, characterized in that: The step (2) includes: (a) Extracting the texture structure of the text region in the local branch. This part of the task includes learning the binary segmentation task of the text region mask and training the network using pyramid soft mask supervision in the horizontal and vertical directions; (b) A global pyramid mask is learned for the bounding boxes of all aligned non-empty cells in the global branch.

3. A complex table structure recognition method for scientific literature according to claim 1, characterized in that: The step (6) includes: (a) First, the coordinates of the text box have been obtained in the previous steps, and the right edge of the text box is given priority as the vertical dividing line; (b) Check whether the vertical dividing line passes through the area between the current character and the previous character. If so, it means that the two text elements do not belong to the same cell; (c) Traverse each text block, determine the column position of the text block in the table, then add it to the corresponding position in the table, and finally return a list containing table objects.

4. The complex table structure recognition method for scientific literature according to claim 1, characterized in that: The step (7) comprises: (a) First determine the number of rows and columns in the table object list and check whether there are any merged cells; (b) Determine whether there is a cross-cell situation by traversing the table row and column structure; (c) If there are cross-cell situations, use rowspan and colspan tags to restructure into HTML format and finally form the final table structure.

Images