Table structure recognition method based on transformer row-column attention mechanism

Abstract

Disclosed in the present invention is a table structure recognition method based on a Transformer row-column attention mechanism, the method comprising: S1, by means a partitioning mode with a fixed width and height, partitioning a table image into multiple rows and columns of patches; S2, flattening the two-dimensionally arranged patches which are obtained by partitioning into one-dimensional patches; S3, processing the one-dimensional patches into image vectors and performing linear projection, and simultaneously concatenating one-dimensional positional embeddings of the patches with [CLS] vectors, and then using a concatenated result as a model input; S4, performing attention mechanism-based feature extraction: excluding, by means of a row-column mask matrix, patches that are not in a same row / column, so as to ensure that computation for features at each position only involves features from the same row and column, using a row-column attention mechanism and a global attention mechanism for table headers to obtain a final matrix, then extracting image feature vectors, and applying a feature pyramid to the features, so as to obtain multi-scale features; and S5, using an object detection model to perform object detection on a feature map, thus finally obtaining table structure information.

Description

A table structure recognition method based on a Transformer row-column attention mechanism TECHNICAL FIELD

[0001] The present application relates to the technical field of image recognition, in particular to a table structure recognition method based on a Transformer row-column attention mechanism. BACKGROUND

[0002] In recent years, it has been a hot research topic in the field to recognize table images using computer vision and deep learning technologies. For example, the invention disclosed in CN116259064A discloses a table structure recognition method and a training method and device for a table structure recognition model. The relative position and hierarchical structure information of the grid lines are obtained by using an attention mechanism to obtain the grid line information and image features. However, this case relies on the table lines and lacks learning of the cell text features, so the recognition ability for wireless tables will decrease. For example, the invention disclosed in CN117237968A discloses a table structure recognition method and system based on a Transformer row-by-row scanning method. The decoding module is designed according to the two-dimensional form of the table logic, and the table sequence is generated using a semi-autoregressive form. However, this case mainly relies on the features of the rows and lacks feature extraction on the columns.

[0003] In addition, foreign scholars have also studied table structure recognition methods. For example, Dosovitskiy A et al. published "An Image is Worth 16x16 Words:Transformers for Image Recognition at Scale", which proposed using a global attention mechanism to learn global features. However, this method causes non-associated region features to be considered as important features, and the difficulty in processing long sequences increases the consumption of computing resources. For another example, Beltagy I et al. published "Longformer:The Long-Document Transformer", which mentioned a local attention mechanism, but the fixed window mechanism cannot adapt to the changes in different importance regions in table images.

[0004] The main problem of the above-mentioned prior art is that when processing table images, the row and column relationship of the table cannot be fully utilized to improve the attention accuracy of the model, the cell text features are not fully utilized, resulting in a decrease in the recognition ability for wireless tables, and the structural features of the table are not fully utilized to reduce the attention focus area. SUMMARY

[0005] To solve the above problems, the application provides a table structure recognition method based on a Transformer row and column attention mechanism (RCAM), which combines the row and column structure characteristics of the table and the cell information, and the row and column attention mechanism ensures that the features of each position only interact with the features of the same row and column through a row and column mask matrix during attention calculation, thereby improving the processing efficiency and accuracy of table data.

[0006] The application provides an attention calculation method of the row and column attention mechanism (RCAM), and the core is the design of a row and column mask matrix, so that the feature position can only interact with the features of the same row and column during attention calculation, thereby enhancing the recognition ability of the model for the table structure. Meanwhile, in order to emphasize the influence of the table header on the entire table structure, global attention calculation for the table header information is added, thereby improving the importance and recognition accuracy of the model for the table header information.

[0007] To achieve the above purpose, the application provides a table structure recognition method based on a Transformer row and column attention mechanism, which comprises the following steps:

[0008] Step S1: dividing a table picture into a plurality of two-dimensional arranged image blocks in multiple rows and multiple columns through a fixed width and height division method;

[0009] Step S2: flattening the divided two-dimensional arranged image blocks into one-dimensional image blocks;

[0010] Step S3: processing the one-dimensional image blocks into image vectors and linearly projecting, simultaneously combining the one-dimensional position embedding of the image blocks and the [CLS] vector, adding the [CLS] vector at the beginning position of the image vector, so that the model can utilize the category information for classification, and then taking the combined result as the input of the model;

[0011] Step S4: feature extraction based on the attention mechanism, excluding the image blocks that are not in the same row / column through a row and column mask matrix, so as to ensure that the features of each position only calculate with the features from the same row and column, using the row and column attention and the global attention mechanism of the table header to obtain the final matrix, then extracting the feature vector of the image, and performing feature pyramid on the features to obtain multi-scale features;

[0012] Step S5: using the extracted features of step S4 to access a target detection model, detecting the target on the feature map, and the output of the target detection result comprises a row detection frame, a column detection frame and a cell detection frame, and finally obtaining the information of the table structure.

[0013] In an embodiment of the present application, step S1 is specifically:

[0014] For a given original image where H, W and C are the height, width and channel number of the image respectively, the image I is divided into P×P non-overlapping image blocks by linear projection, where P is the side length of the image block.

[0015] In an embodiment of the present application, the one-dimensional image block flattened in step S2 is represented as where N×N represents the number of image blocks, and where all image blocks are numbered using numerical values, the numbering length is N×N, and the numbering starts from 1.

[0016] In an embodiment of the present application, the final matrix obtained in step S4 is specifically:

[0017] The row-column mask matrix M is where N×N still represents the number of image blocks, and the specific definition of the row-column mask matrix M is:

[0018] where M ij is the matrix element to be calculated, i is the image block number to be calculated, j is the image block number to be calculated with attention to i, i and j range from (1, N×N), w represents the value of the width of the original image divided by the width of the image block, the symbol represents rounding down, and mod represents congruence;

[0019] 0 represents the value of all rows and columns corresponding to the row and column positions of all image blocks to be calculated in the row-column mask matrix; -1e 10 represents a large negative number to eliminate the weight not in the same row or column;

[0020] Since the table header occupies the first y rows of image blocks, y is a positive integer, so the global attention calculation is performed on the first y rows, that is, all row and column results in the first y rows of the row-column mask matrix M are set to 0 during calculation;

[0021] There is also a vector [CLS] before all image block vectors, which is used to calculate the attention of each row-column image block to obtain the global feature, and the final row-column attention mask matrix is:

[0022] In the formula, M is the row-column mask matrix; Q, K and V are three input representation vectors of the self-attention mechanism, where Q represents the query vector, K represents the key vector, and V represents the numerical vector; d k is the dimension of the vector / hidden layer; represents element-wise addition.

[0023] In an embodiment of the present application, the target detection model in step S5 comprises Mask-RCNN or Cascade R-CNN.

[0024] The table structure recognition method based on the row-column attention mechanism of the Transformer provided by the present application can effectively combine the row-column structure characteristics of the table through the row-column attention mechanism (RCAM), improve the effect of the model on table structure recognition, and better capture the information of the table header through the introduced global row-column attention, which plays a guiding role for the overall structure of the table. The combination of the two can complete the function of table structure recognition. BRIEF DESCRIPTION OF DRAWINGS

[0025] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0026] Fig. 1 is a flowchart of an embodiment of the present application;

[0027] Fig. 2 is a schematic diagram of image block division in an embodiment of the present application;

[0028] Fig. 3 is a schematic diagram of image block tiling in an embodiment of the present application;

[0029] Fig. 4 is a schematic diagram of image block embedding in an embodiment of the present application;

[0030] Fig. 5 is a final matrix obtained by using only row-column attention for feature extraction in an embodiment of the present application;

[0031] Fig. 6 is a final matrix obtained by using a global attention mechanism of the table header for feature extraction in an embodiment of the present application. DETAILED DESCRIPTION

[0032] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0033] Fig. 1 is a table structure recognition method based on the row-column attention mechanism of the Transformer provided by the present application, which comprises the following steps:

[0034] Step S1: image block division, dividing the table picture into a two-dimensional arrangement of image blocks by a fixed width and height size division method;

[0035] In this embodiment, step S1 is specifically:

[0036] For a given original image where H, W and C are the height, width and channel number of the image respectively, the image I is divided into P x P non-overlapping image blocks by linear projection, where P is the image block side length.

[0037] For example, the original image is redefined in size to facilitate the input of the model, and the original image is scaled to an image size of 224 x 224, that is, H and W are both 224, and is divided into 14 x 14 image blocks, obtaining image blocks of 16 x 16 size, that is, P is 16.

[0038] Step S2: image block tiling, flattening the two-dimensional arrangement of image blocks into one-dimensional image blocks; the image blocks obtained in step S1 can be tiled, and the original two-dimensional arrangement of image blocks becomes one-dimensional arrangement of image blocks;

[0039] In this embodiment, the one-dimensional image blocks flattened in step S2 are represented as where N x N represents the number of image blocks, and where all image blocks are numbered using numerical values, the numbering length is N x N, and the numbering starts from 1.

[0040] Continuing with the above example of the image, the number of image blocks is 14 x 14, that is, N is 14.

[0041] Step S3: image block embedding, processing the one-dimensional image blocks into image vectors and linear projection, while merging the one-dimensional position embedding of the image blocks and the [CLS] vector, the [CLS] vector is added at the beginning of the image vector to enable the model to use class information for classification, and then the merged result is used as the input of the model; where the [CLS] vector represents the class information of the entire image;

[0042] For example, the one-dimensional arrangement of image blocks can be represented by image embedding to obtain an image vector, and the [CLS] vector is inserted at the beginning of the image vector for global information classification, and then the image vector containing the [CLS] vector is copied by row to obtain a vector matrix.

[0043] Step S4: feature extraction based on attention mechanism, excluding non-same row / column image blocks through row-column mask matrix to ensure that the feature of each position is only calculated with the features from the same row and column, thereby limiting the attention range and reducing the calculation burden, using row-column attention and global attention mechanism of table header to obtain the final matrix, then extracting the feature vector of the image, and performing feature pyramid on the features to obtain multi-scale features;

[0044] In the embodiment, the specific process of obtaining the final matrix in step S4 is as follows:

[0045] The row-column mask matrix M is Where N x N still represents the number of image blocks, and the specific definition of the row-column mask matrix M is as follows:

[0046] Where M ij is the matrix element to be calculated, i is the image block number to be calculated, j is the image block number to be calculated with i, i and j range from (1, N x N), w represents the value of the width of the original image divided by the width of the image block, which can be referred to as two-dimensional feature width; the symbol represents rounding down, mod represents congruence, and j≡j(mod w) means that i and j have the same remainder when divided by w;

[0047] 0 represents the value of all rows and columns corresponding to the row and column positions of all to-be-calculated image blocks in the row-column mask matrix; -1e 10 represents a large negative number to eliminate the weight of the image blocks not in the same row or column;

[0048] Considering that the table header hides the overall structure of the table, when global attention is introduced at the table header position, since the table header may occupy the first y rows of image blocks, global attention is calculated for the first y rows, that is, all row and column results of the first y rows in the row-column mask matrix M are set to 0, where y is a positive integer which can be set according to the specific use scenario, and the specific value is not limited in the embodiment; note that there is a vector [CLS] before all image block vectors, which is used to calculate the attention of each row and column image block to obtain global features, and finally the row-column attention mask matrix is obtained as follows:

[0049] In the formula, M is the row-column mask matrix; Q, K, and V are three input representation vectors of the self-attention mechanism, where Q represents the query vector, K represents the key vector, and V represents the value vector, which are obtained from the original image block vector through linear transformation; k is the dimension of the vector / hidden layer, and is selected which can make Q x K conform to the distribution of N(0, 1), similar to normalization; denotes element-wise addition. The row-column mask matrix M of this embodiment needs to be used after multi-head attention calculation and before the Softmax operation.

[0050] Step S5: detecting the head prediction result, i.e. using the extracted features in step S4 to access the target detection model to detect the target on the feature map, and the result of target detection needs to output three results, respectively, as row detection frame 101, column detection frame 102 and cell detection frame 103, and finally obtain the information of the table structure. The target detection model may be, for example, Mask-RCNN or Cascade R-CNN, but is not limited thereto.

[0051] Fig. 2 is a schematic diagram of image block division according to an embodiment of the present application, Fig. 3 is a schematic diagram of image block tiling according to an embodiment of the present application, Fig. 4 is a schematic diagram of image block embedding according to an embodiment of the present application, Fig. 5 is a final matrix obtained by using only row-column attention for feature extraction according to an embodiment of the present application, and Fig. 6 is a final matrix obtained by using row-column attention table header global attention mechanism for feature extraction according to an embodiment of the present application. In order to clearly illustrate the foregoing process, referring to Figs. 2-6 in combination with Fig. 1, the attention calculation of the second row and the second column is described in detail as follows taking the number of image blocks as 3x3 as an example:

[0052] Firstly, the given original table image is divided into 3x3 two-dimensional image blocks as shown in Fig. 2;

[0053] Then, the image blocks are tiled into one-dimensional image blocks, each image block corresponds to a number, such as 1-9 for 9, and then the embedding operation is performed to obtain an image vector, i.e. flattened into an image vector, and a [CLS] vector mark is added before the image vector, i.e. 3x3+1=10 vectors, as shown in Fig. 3;

[0054] Then, the combined vector is copied 3x3+1 times to form a 10x10 matrix, i.e. a vector matrix, as shown in Fig. 4. This matrix is used as the input of the model. Note that each image block is processed as a vector;

[0055] Next, since the attention calculation is performed on the second row and the second column, i.e. on the image block with the serial number 5, the row-column attention calculation needs to combine the row and the column where the image block is located. Referring to formula (1) and Fig. 2, the image block serial numbers corresponding to the row are 4 and 6, the image block serial numbers corresponding to the column are 2 and 8, and the global mark CLS. The values of the positions of the row-column mask matrix M corresponding to these serial numbers are set to 0, as shown in the light gray positions in Fig. 5;

[0056] Suppose the table header occupies the first 3 rows, i.e. the global row-column attention of the first 3 rows is introduced, then the 3 rows except the first row will participate in the calculation, and the complete calculation position is shown in the light gray part in Fig. 6;

[0057] In FIG. 6, the dark gray area represents the self-attention calculation part, the light gray is the global row-column attention calculation, and the gray is the row-column attention calculation.

[0058] The table structure recognition method based on the Transformer row-column attention mechanism provided by the application can effectively combine the row-column structure characteristics of the table through the row-column attention mechanism (RCAM), improve the effect of the model on table structure recognition, and better capture the information of the table header through the introduced global row-column attention, which plays a guiding role for the overall structure of the table. The combination of the two can complete the function of table structure recognition.

[0059] Table 1 is an experiment on the method of the application using the PubTables-1M dataset to evaluate the performance of the designed model. The standard object detection metrics AP, AP50, AP75, AR and table context precision matching accuracy Acc Con are used for evaluation, where Acc Con The index considers the proportion of tables that accurately match all cells (even blank cells). The Vit scheme in the background art (An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale) and the TATR (PubTables-1M: Towards comprehensive table extraction from unstructured documents) scheme proposed for PubTables-1M are compared. It is obvious that RCAM performs better in each index.

[0060] Table 1

[0061] In addition, in order to verify the influence of the table header on RCAM, a group of experiments are designed, as shown in Table 2, when the current n-row image block is the table header, the influence of the entire table structure recognition on the performance. It can be found that for PubTables-1M data, the best n value is 3, and for actual scenarios, the specific n value needs to be measured.

[0062] Table 2

[0063] Those skilled in the art can understand that the drawings are only schematic diagrams of an embodiment, and the modules or processes in the drawings are not necessarily necessary for implementing the application.

[0064] Those skilled in the art can understand that the modules in the device in the embodiments can be distributed in the device in the embodiments as described in the embodiments, or can be changed to be located in one or more devices different from the embodiments. The modules in the above embodiments can be combined into one module, or can be further split into multiple sub-modules.

[0065] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A table structure recognition method based on a Transformer row-column attention mechanism, characterized in that, The method comprises the following steps: Step S1: dividing the table picture into a two-dimensional arrangement of image blocks in multiple rows and multiple columns by a fixed width and height segmentation method; Step S2: flattening the two-dimensional arrangement of image blocks into one-dimensional image blocks; Step S3: processing the one-dimensional image blocks into image vectors and linearly projecting them, while merging the one-dimensional position embedding of the image blocks and the [CLS] vector, the [CLS] vector being added at the beginning of the image vector to enable the model to utilize class information for classification, and then taking the merged result as the input of the model; Step S4: feature extraction based on an attention mechanism, excluding image blocks that are not in the same row / column through a row-column mask matrix to ensure that the features at each position are only calculated with features from the same row and column, using row-column attention and global attention mechanism of the table header to obtain the final matrix, then extracting the feature vector of the image, and performing feature pyramid on the features to obtain multi-scale features; Step S5: using the extracted features in step S4 to access a target detection model to detect targets on the feature map, the output of the target detection result including row detection boxes, column detection boxes and cell detection boxes, and finally obtaining the information of the table structure.

2. The table structure recognition method based on the Transformer row-column attention mechanism according to claim 1, characterized in that, Step S1 is specifically: For a given original image Wherein H, W and C are the height, width and channel number of the image respectively, the image I is divided into P*P non-overlapping image blocks through linear projection, wherein P is the side length of the image block.

3. The table structure recognition method based on the Transformer row-column attention mechanism according to claim 1, characterized in that, The flattened one-dimensional image block is denoted as where N x N denotes the number of image blocks, and where all image blocks are numbered using numerical values, the numbering length is N x N, and the numbering starts from 1.

4. The table structure recognition method based on the Transformer row-column attention mechanism according to claim 1, characterized in that, The specific process of obtaining the final matrix in step S4 is: The matrix mask matrix M is where N x N still represents the number of image blocks, and the specific definition of the row-column mask matrix M is: where M ij is the matrix element to be calculated, i is the image block number to be calculated, j is the image block number to which attention needs to be calculated with i, i and j range from (1, N x N), w represents the value of the width of the original image divided by the width of the image block, the symbol represents rounding down, and mod represents congruence; 0 represents all the values in the row and column mask matrix corresponding to the row and column position where the image block to be calculated is located; -1e 10 represents a large negative number to eliminate weights that are not in the same row or column; Since the table header occupies the image blocks in the first y rows, y is a positive integer, so the global attention calculation is performed on the first y rows, specifically, all row and column results of the first y rows in the row-column mask matrix M are set to 0 when calculating; There is also a vector [CLS] before all image block vectors, which is used to calculate the attention of each row and column image block to obtain global features, and finally the row and column attention mask matrix is obtained as: In the formula, M is a row-column mask matrix; Q, K, and V are respectively three input representation vectors of the self-attention mechanism, wherein Q represents a query vector, K represents a key vector, and V represents a value vector; d k is the dimension of the vector / hidden layer; represents element-wise addition.

5. The table structure recognition method based on the Transformer row-column attention mechanism according to claim 1, characterized in that, The target detection model in step S5 includes Mask-RCNN or Cascade R-CNN.

Images