Table structure recognition method and system based on row-by-row scanning of transformer

Abstract

The application provides a table structure recognition method and system based on row-by-row scanning of a Transformer, and relates to the field of table image recognition.A new table structure recognition model is designed, which is composed of an encoder module, a row decoder module and a cell decoder module, combines the unique characteristics of the table itself, decodes in a two-dimensional form in accordance with the table logic, and generates a table sequence by using a semi-autoregressive form, so as to relieve the serious error accumulation problem caused by a long table sequence to a certain extent.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of table image recognition, in particular to a table structure recognition method and system based on line-by-line scanning of Transformer. BACKGROUND

[0002] Table parsing and understanding is a research hotspot in recent years, and table structure recognition is an important part of the whole process, which converts tables that computers cannot directly parse, such as pictures, into formats that computers can directly edit. With the development of deep learning, this field has made rapid progress. Inspired by related work in multi-modal fields such as image description and visual question answering, current table structure recognition gradually begins to try to use multi-modal generative methods based on encoder-decoder structure. The encoder extracts rich visual features through convolutional neural networks and Transformer, and the decoder obtains the required features through attention mechanism, and predicts each character in the sequence according to the pre-defined text sequence order.

[0003] Current table structure recognition models based on multi-modal generative methods use autoregressive form to generate table sequences. This scheme generates the next required text character by character during decoding, which will cause serious error accumulation. Existing methods usually use original HTML code to describe table structure and as the final output form, but one-dimensional HTML code does not match two-dimensional table pictures in terms of spatial position and logical relationship. SUMMARY

[0004] The purpose of the present application is to provide a table structure recognition method based on line-by-line scanning of Transformer, which designs a new table structure recognition model, combines the unique characteristics of the table itself, and decodes in a two-dimensional form that conforms to the table logic. At the same time, by using semi-autoregressive form to generate table sequences, the problem of serious error accumulation caused by long table sequences is alleviated to some extent.

[0005] To achieve the above purpose, the present application adopts the following technical solutions:

[0006] A table structure recognition method based on line-by-line scanning of Transformer, comprising the following steps:

[0007] 1) input the table picture into the encoder module, the encoder module uses residual convolutional network to extract visual features from the table picture; and uses Transformer encoder to extract global semantic information from the visual features to generate visual semantic features;

[0008] 2) input the visual semantic features into a row decoder module, the row decoder module uses a Transformer layer to interact with the visual semantic features using a cross-attention mechanism to predict row information of the table;

[0009] 3) input the visual features, the row information and the visual semantic features into a cell decoder, the cell decoder uses a Transformer layer to fuse the visual features, the row information and previously predicted cell information as input, and interacts with the visual semantic features, and predicts cell information in each row in a row-by-row manner from top to bottom;

[0010] 4) combine the predicted row information and the cell information to generate a cell logical structure and corresponding position information.

[0011] Further, the encoder module in step 1) divides the visual features into multiple visual blocks of equal size, and then a Transformer encoder interacts with all the visual blocks through a self-attention mechanism to extract global semantic information.

[0012] Further, the row decoder module in step 2) includes multiple layers of stacked Transformer layers and two independent three-layer fully connected layers, each Transformer layer is composed of a self-attention layer, a cross-attention layer and a fully connected layer; the Transformer layer uses the embedding vector converted by the row index as input, uses a cross-attention mechanism to interact with the visual semantic features to generate a high-dimensional feature vector sequence corresponding to the embedding vector; then two independent three-layer fully connected layers are used to predict the row information of the table based on the high-dimensional feature vector sequence, the row information includes the category and position of each row.

[0013] Further, the row decoder module in step 2) is pre-trained, and when training, the predicted content is matched with the labeled information through the Hungarian matching algorithm, and the prediction loss of the row category and position is used to back-propagate the model parameters.

[0014] Further, the row decoder module in step 3) includes multiple layers of stacked Transformer layers and four independent three-layer fully connected layers, each Transformer layer is composed of a self-attention layer, a cross-attention layer and a fully connected layer, the Transformer layer uses a cross-attention mechanism to interact with the visual semantic features to generate a high-dimensional feature vector sequence; then four independent three-layer fully connected layers are used to predict the cell information of the table based on the high-dimensional feature vector sequence, the cell information includes the position, row span number, column span number and cell category of the cell.

[0015] Further, the input of the row decoder module in step 3) includes historical embedding vectors, future embedding vectors and current embedding vectors, wherein the historical embedding vectors represent the past predicted row and cell information, the future embedding vectors represent the currently un-predicted row and cell information, and the current embedding vectors represent the currently required predicted row and cell information; the historical embedding vectors, the future embedding vectors and the current embedding vectors are respectively composed of three parts of type encoding, visual clues and cell index.

[0016] Further, the type encoding is a learnable embedding vector used to mark the type to which the current input vector belongs, and the type includes four types of <historical embedding vector, row>, <historical embedding vector, cell>, <current embedding vector, row> and <future embedding vector, row>.

[0017] Further, the visual clue is based on visual features, for the embedding vector of a row, the visual clue is mapped into the visual features using the area surrounded by the upper and lower boundaries of the corresponding row, and is encoded into a fixed size visual embedding using ROIAlign; for the embedding vector of a cell, the visual clue is mapped into the visual features using the area surrounded by the four edges of the corresponding cell, and is encoded into a visual embedding of the same size using ROIAlign.

[0018] Further, the cell index uses a learnable embedding vector as input.

[0019] Further, the cell decoder module in step 3) is trained in advance, and during the training, the predicted content is matched with the labeled information by using the Hungarian matching algorithm, and the model parameters are trained by using the prediction loss of the cell position, the number of row spans, the number of column spans and the cell category and back propagation.

[0020] A table structure recognition system based on row-by-row scanning of the Transformer, comprising:

[0021] An encoder module, comprising a residual convolutional network and a Transformer encoder, wherein the residual convolutional network is used to extract visual features from the input table picture, and the Transformer encoder is used to extract global semantic information from the visual features and generate visual semantic features;

[0022] A row decoder module mainly comprising multiple layers of Transformer, each layer of Transformer being composed of a self-attention layer, a cross-attention layer and a fully connected layer, and being used to interact with the visual semantic features using the cross-attention mechanism to predict the row information of the table;

[0023] The cell decoder module mainly comprises multi-layer Transformer layers, each layer of the Transformer being composed of a self-attention layer, a cross-attention layer and a full connection layer, and is used for fusing visual features, line information and previously predicted cell information as input, interacting with visual semantic features, and predicting cell information in each line in a line-by-line top-down manner.

[0024] The technical solution of the present application has the following advantages:

[0025] 1. Line-by-line top-down prediction method: The line-by-line top-down scanning method adopted by the present application is more consistent with the structural characteristics of the table, which is consistent with the traditional two-dimensional table organization form (organized by rows and columns), and is more in line with human reading and processing habits.

[0026] 2. Alleviate error accumulation problem: By adopting the semi- autoregressive method to predict the cell structure, the number of iteration steps required for module prediction is reduced, which to some extent alleviates the error accumulation problem that may occur when processing long sequences.

[0027] 3. Stable model performance: The present application performs well on existing data sets, especially when processing large or complex tables, the performance is particularly stable and excellent.

[0028] 4. No need for complex post-processing: The prediction order of the present application is more in line with the organization logic of the table, which makes it unnecessary to perform complex post-processing steps after output, simplifying the process.

[0029] 5. Combine the advantages of Transformer: The present application makes full use of the long-distance dependence capturing ability of Transformer, making it more efficient and accurate when processing tables with complex structures.

[0030] 6. Reduce the number of iteration steps required for prediction: Compared with the traditional fully autoregressive model, the semi-autoregressive model of the present application can predict more information at each step, thereby reducing the number of iteration steps required for prediction and improving efficiency.

[0031] 7. New recognition model design: The present application proposes a new table structure recognition model that combines the characteristics of the table itself, which can capture local details and understand the overall structure. BRIEF DESCRIPTION OF DRAWINGS

[0032] Figure 1 It is a table structure recognition system structure diagram based on the line-by-line scanning of the Transformer proposed by the present application.

[0033] Figure 2 It is a visualization diagram of the matching degree of the prediction result and the annotation. DETAILED DESCRIPTION

[0034] In order to make the technical features and advantages or technical effects of the above technical solutions of the present application more obvious and easy to understand, the following will be described in detail with reference to the drawings.

[0035] The present application provides a table structure recognition method and system based on row-by-row scanning of Transformer, using a semi-autoregressive form table structure recognition model that scans row by row. The input of the system is a table picture, and the output is the complete HTML code corresponding to the current table picture. As shown in the figure, the system consists of three parts: an encoder module, a row decoder module, and a cell decoder module. Figure 1

[0036] The encoder module consists of a residual convolutional network and a Transformer encoder. The residual convolutional network is used to extract rich visual features from the table picture, and the residual connection is used to ensure that the module still has good performance when the number of layers is deep. After the table picture passes through the residual convolutional network, it can obtain good visual features. Then the obtained visual features are divided into multiple visual blocks of the same size and input into the Transformer encoder. The Transformer encoder interacts with all visual blocks through the self-attention mechanism to extract more global semantic information, i.e., visual semantic features. It can be seen that the encoder module obtains visual semantic features through the residual convolutional network and the Transformer encoder, which are used in the subsequent decoding process.

[0037] The decoding process of the model is responsible for the execution of the row decoder module and the cell decoder module, both of which use the Transformer decoder as the main structure.

[0038] The row decoder module includes multiple layers of stacked Transformer layers and two independent three-layer fully connected layers. Each Transformer layer is composed of a self-attention layer, a cross-attention layer, and a fully connected layer. The row index of the row decoder module uses a learnable embedding vector as input, and then uses the cross-attention mechanism to interact with the visual semantic features. After the calculation of multiple Transformer layers, a high-dimensional feature vector sequence is obtained, which is one-to-one corresponding to the number and order of the input learnable embedding vector. Finally, these high-dimensional feature vector sequences pass through two independent three-layer fully connected layers to predict two prediction sets, representing the class and position prediction of each row in the table. The training stage of the row decoder module is similar to DETR, using the Hungarian algorithm to match each prediction with the labeled information, and using the backpropagation algorithm to calculate the gradient of the loss function with respect to the model parameters.

[0039] ​The cell decoder module is different from the row decoder module only in the classification header, and the rest of the model structure is completely consistent, only the input and output forms are modified. That is, the cell decoder module is also stacked by multiple layers of Transformer, and each layer is composed of a self-attention layer, a cross-attention layer, and a full connection layer. The cell decoder adopts a semi-autoregressive form for decoding, and is predicted in order from top to bottom according to the order of the row to which the cell belongs. In each iteration step, the cell decoder module predicts all the cells of the current row. The input of the cell decoder module is divided into three parts: historical embedding vectors, future embedding vectors, and current embedding vectors. Among them, the historical embedding vectors are used to represent the row and cell information that has been predicted in the previous iteration process, that is, all the row and cell information above the current required prediction row; the future embedding vectors are used to represent the row information that has not yet been predicted, that is, all the row information below the current required prediction row; and the current embedding vectors are used to represent the relevant information of the current required prediction row.

[0040] The embedding vector of each input of the cell decoder module is composed of three parts, namely type code, visual clue and cell index. Among them, the type code is a learnable embedding vector, which is used to mark the type to which the current input vector belongs, and contains four types: < historical embedding vector, row >, < historical embedding vector, cell >, < current embedding vector, row > and < future embedding vector, row >. The visual clue is used to integrate high-level visual information to speed up the convergence speed and final effect of the model. All visual clues come from the visual features extracted by the residual convolutional network in the encoder module. For the embedding vector of type "row", the visual clue corresponding to it uses the area surrounded by the upper boundary and the lower boundary of the corresponding row, maps this area to the visual features output by the residual convolutional network, and uses ROIAlign (Region of Interest Align, a commonly used feature extraction method) to encode the features of this area into a fixed size visual embedding. For the embedding vector of type "cell", the visual clue corresponding to it uses the area surrounded by the four edges of the corresponding cell, and also uses ROIAlign to extract the visual features of the corresponding area and encode them into a visual embedding of the same size. The cell index is similar to the input of the row decoder module, which uses a learnable embedding vector as input to mark different input vectors. The cell decoder module also uses the cross-attention mechanism to interact with the visual semantic features extracted by the encoder module. After the calculation of multiple Transformer layers, a high-dimensional feature vector sequence is predicted. Finally, the obtained high-dimensional feature vector sequence passes through four different three-layer fully connected layers to predict the position of the cell, the number of row spans, the number of column spans and the cell category. Similar to the row decoder module, the cell decoder module also uses the Hungarian algorithm to match the final prediction result with the labeled information during the training stage, and performs backpropagation training, that is, uses the backpropagation algorithm to calculate the gradient of the loss function with respect to the model parameters.

[0041] Based on the above model architecture, a table structure recognition method based on Transformer is proposed, which can be divided into the following steps:

[0042] 1. The input table picture is extracted by the residual convolutional network of the encoder module to extract visual features. The extracted visual features are then input into two decoder modules after a Transformer encoder extracts global semantic information to generate visual semantic features.

[0043] 2. The row decoder module adopts a Transformer decoder as the main structure, takes a learnable embedding vector as input, interacts with the visual semantic features extracted by the encoder module using a cross-attention mechanism, and predicts all row information, such as the position and category of the row.

[0044] 3. The cell decoder module adopts a Transformer decoder as the main structure, takes the row information predicted by the row decoder module, the cell information predicted by the cell decoder in the previous iteration process, and the visual features extracted by the residual convolutional network as input, and interacts with the visual semantic features extracted by the encoder module, and predicts the cell information in each row in the order from top to bottom, including the position, row span number, column span number, and cell category.

[0045] 4. The row information and cell information predicted are combined to generate the required cell logical structure and corresponding position information.

[0046] 5. The row decoder module and the cell decoder module match the predicted content with the labeled information through the Hungarian matching algorithm, and use the prediction loss of the category and position to back-propagate the model parameters.

[0047] Experimental test:

[0048] Extensive experiments were conducted to evaluate the effect of the technical solution of the present application on the table structure recognition task. The present application tests on three data sets: PubTabNet is a large-scale table data set containing 500,777 training set data and 9,115 validation set data. The data of PubTabNet comes from the tables in scientific papers, and the logical structure of the table and the text in the table are given using HTML. The data of SciTSR comes from scientific literature in PDF form, and the table structure is given by LaTeX analysis, containing 12,000 training data and 3,000 test data. At the same time, the data set authors selected 716 pictures from SciTSR to form the SciTSR-COMP test set to further evaluate the performance of the model on complex tables. SynthTabNet is a large-scale generated data set containing 600,000 pictures and labels, and all pictures are divided into training set, validation set and test set according to the ratio of 80%, 10% and 10%.

[0049] Table 1 shows the performance of the present application on PubTabNet, the present application achieves 97.8% on TEDS evaluation index and 98.9% on TEDS-Struct index. Table 2 shows the performance of the present application on SciTSR, in SciTSR test set, the present application achieves comparable performance with other mainstream methods, and in SciTSR-COMP test set, the present application achieves 0.3% improvement compared with mainstream methods. Table 3 shows the performance of the present application on SynthTabNet, compared with mainstream methods, the present application also achieves 2% improvement on TEDS evaluation index. These results show that the present application achieves excellent performance in various different data sets.

[0050] Table 4 evaluates the performance of different methods when facing tables of different sizes. PubTabNet validation set is divided into 3 groups according to the length of output HTML, that is, the smallest 1 / 3 of the table is divided into PubTabNet-small, the largest 1 / 3 of the table is divided into PubTabNet-large, and the remaining table is divided into PubTabNet-medium. In the results shown in Table 4, as the size of the table increases, the performance of all image-to-text table structure recognition methods decreases, but when comparing the performance decline gap between different divisions, the performance of the present application is more stable and the decline is smaller as the size of the table increases.

[0051] Figure 2 The performance of different methods in matching the predicted cell position with the labeled position on PubTabNet validation set is shown. TableMaster also uses image-to-text form for table structure recognition, and the prediction of this method shows a clear trend in spatial position: the prediction accuracy of cells close to the right and close to the bottom is lower. The prediction result of the present application maintains greater stability in the horizontal direction, because the present application predicts cells in a row-by-row from top to bottom manner, thus alleviating the error accumulation problem of the model in the horizontal direction.

[0052] Table 1 Experimental results of the present application and other methods on PubTabNet

[0053] Method TEDS (%) TEDS-Struct (%) EDD 88.3 - TableFormer 93.6 96.8 TableMaster 96.8 - LGPMA 94.6 96.7 FLAG-Net 95.1 - RobusTabNet - 97.0 TSRFormer - 97.5 DRCC (the present invention) 97.8 98.9

[0054] Table 2 Experimental results of the present application and other methods on SciTSR

[0055] Method SciTSR SciTSR-COMP LGPMA 98.8 98.0 FLAG-Net 99.5 98.5 RobusTabNet 99.3 98.7 TSRFormer 99.4 98.9 DRCC (the present invention) 99.5 99.2

[0056] Table 3 Experimental results of the present application and other methods on SynthTabNet

[0057]

[0058] Table 4 Experimental results of the present application and other methods on PubTabNet with different size partitions

[0059] Method PubTabNet-small PubTabNet-medium PubTabNet-large EDD 91.79 90.50 87.42 TableMaster 97.35 96.58 94.61 DRCC (the present invention) 98.36 97.86 97.25

[0060] Although the present application has been disclosed with examples as above, it is not intended to limit the present application, and any appropriate modification or equivalent replacement made by those skilled in the art to the technical solutions of the present application shall be covered within the protection scope of the present application, and the protection scope of the present application is defined by the claims.

Claims

1. A method for recognizing a table structure based on a row-by-row scanning of a Transformer, characterized in that, The method comprises the following steps: 1) inputting a table picture into an encoder module, the encoder module extracting visual features from the table picture using a residual convolutional network; and using a Transformer encoder to extract global semantic information from the visual features to generate visual semantic features; 2) inputting the visual semantic features into a row decoder module, the row decoder module using a Transformer layer to interact with the visual semantic features using a cross-attention mechanism to predict row information of the table; 3) inputting the visual features, row information and visual semantic features into a cell decoder, the cell decoder using a Transformer layer to fuse the visual features, row information and previously predicted cell information as input, and interacting with the visual semantic features to predict cell information in each row in a row-by-row manner from top to bottom; 4) combining the predicted row information and cell information to generate cell logical structures and corresponding position information.

2. The method of claim 1, wherein, The encoder module in step 1) divides the visual features into multiple visual blocks of equal size, and then the Transformer encoder interacts with all the visual blocks through a self-attention mechanism to extract global semantic information.

3. The method of claim 1, wherein, The row decoder module in step 2) comprises multiple layers of stacked Transformer layers and two independent three-layer fully connected layers, each Transformer layer being composed of a self-attention layer, a cross-attention layer and a fully connected layer; the Transformer layer uses a cross-attention mechanism to interact with the visual semantic features using an embedding vector converted from a row index as input to generate a high-dimensional feature vector sequence corresponding to the embedding vector; then two independent three-layer fully connected layers are used to predict row information of the table based on the high-dimensional feature vector sequence, the row information including the category and position of each row.

4. The method of claim 1 or 3, wherein, The row decoder module in step 2) is pre-trained, and during training, the predicted content is matched with the labeled information through a Hungarian matching algorithm, and the model parameters are trained through back propagation using the prediction loss of the row category and position.

5. The method of claim 1, wherein, The row decoder module in step 3) comprises multiple layers of stacked Transformer layers and four independent three-layer fully connected layers, each Transformer layer being composed of a self-attention layer, a cross-attention layer and a fully connected layer, the Transformer layer using a cross-attention mechanism to interact with the visual semantic features to generate a high-dimensional feature vector sequence; then four independent three-layer fully connected layers are used to predict cell information of the table based on the high-dimensional feature vector sequence, the cell information including the position, row span number, column span number and cell category of the cell.

6. The method of claim 1 or 5, wherein, The input of the row decoder module in step 3) includes a history embedding vector, a future embedding vector and a current embedding vector, wherein the history embedding vector represents the past predicted row and cell information, the future embedding vector represents the currently un-predicted row and cell information, and the current embedding vector represents the currently required predicted row and cell information; the history embedding vector, the future embedding vector and the current embedding vector are respectively composed of three parts of type encoding, visual clue and cell index.

7. The method of claim 6, wherein, The type encoding is a learnable embedding vector used to mark the type to which the current input vector belongs, and the type includes four types of <history embedding vector, row>, <history embedding vector, cell>, <current embedding vector, row> and <future embedding vector, row>; the cell index uses a learnable embedding vector as input.

8. The method of claim 6, wherein, The visual clue is based on visual features, for the embedding vector of a row, the visual clue is mapped to the visual features using the area surrounded by the upper and lower boundaries of the corresponding row, and is encoded into a fixed size visual embedding using ROIAlign; for the embedding vector of a cell, the visual clue is mapped to the visual features using the area surrounded by the four edges of the corresponding cell, and is encoded into a visual embedding of the same size using ROIAlign.

9. The method of claim 1 or 5, wherein, The cell decoder module in step 3) is trained in advance, and during the training, the predicted content is matched with the labeled information by using the Hungarian matching algorithm, and the model parameters are trained by using the prediction loss of the cell position, the number of row spans, the number of column spans and the cell category and back propagation. 10.A Transformer-based table structure recognition system for line-by-line scanning, characterized in that, It comprises: an encoder module comprising a residual convolutional network and a Transformer encoder, wherein the residual convolutional network is used to extract visual features from an input table picture, and the Transformer encoder is used to extract global semantic information from the visual features and generate visual semantic features; a row decoder module mainly comprising multiple layers of Transformer, each layer of Transformer comprising a self-attention layer, a cross-attention layer and a fully connected layer, and being used to interact with the visual semantic features using the cross-attention mechanism and predict the row information of the table; a cell decoder module mainly comprising multiple layers of Transformer, each layer of Transformer comprising a self-attention layer, a cross-attention layer and a fully connected layer, and being used to fuse the visual features, the row information and the past predicted cell information as input, and interact with the visual semantic features to predict the cell information in each row in the order of rows from top to bottom.

Images