A table structure recognition method based on edge convolution interaction

Abstract

This invention discloses a table structure recognition method based on edge convolution interaction. The method constructs a table structure recognition model based on edge convolution interaction: Inputting a table image, a feature map of the table, and the content and position information of the text rows in the table, the method obtains the text center position and samples the corresponding location on the feature map. The sampled image features and the text row position information are concatenated to obtain image features containing the text positions. Using text rows as vertices and the relationships between text rows as edges, the table is represented using a network graph. Edge features between each vertex and its M nearest neighbors are calculated based on edge convolution, and then the vertex and the M edge features are interactively aggregated. Based on the feature vectors obtained after the interactive aggregation of each vertex, the relationships between vertices are classified to obtain the relationships between the text rows in the table. After training the table structure recognition model based on edge convolution interaction, the method is used to accurately recognize the table structure.

Description

Technical Field

[0001] This invention belongs to the field of table structure recognition, and relates to a table structure recognition method based on edge convolution interaction. Background Technology

[0002] Tables in text provide an intuitive and natural way to present data in a format that is easy for humans to understand. Table structure refers to the row and column distribution and logical relationships between cells, including titles, headers, rows, and columns. Table structure recognition can better understand the relationships between cells in a table, facilitating tasks such as generating table text descriptions and table-based question answering. Table structure recognition is a fundamental task in document processing, combining physical and logical layout recognition. It also includes analyzing or recognizing complex tables, aiming to extract data and structural information from them, obtaining the distribution of rows and columns and the logical structure between cells. Due to its importance and difficulty, table structure recognition has attracted a large number of researchers to contribute to this field.

[0003] In recent years, with the development of deep learning technology and influenced by table recognition competitions and public datasets, deep learning methods have been increasingly applied to table structure recognition tasks. Existing deep neural network-based methods can be mainly divided into two categories: one is object detection-based methods, such as those proposed by research teams at Mannheim University of Applied Sciences and the Technical University of Kaiserslautern in Germany, which treat table structure recognition as an object detection problem and introduce deformable convolutions to adapt to the detection of regions of different sizes and shapes; the other is segmentation-based methods, such as those proposed by the National University of Science and Technology in Islamabad, Pakistan, which proposes using recurrent convolutional networks to recognize table structures, taking into account the characteristic of repeated sequences in rows and columns of cells. This method uses two independent models for row and column segmentation, with the center line of the segmented region as the segmentation result.

[0004] While these methods have achieved some performance improvements in table structure recognition tasks, they still cannot accurately determine the relationships between cells. Object detection-based methods cannot accurately determine cell positions when the table image is tilted, the background is complex, or the table has few lines, thus reducing the accuracy of table structure recognition. Segmentation-based methods are limited to the form of the cells containing table objects. The structure of tables is complex and diverse. In order to achieve a concise and aesthetically pleasing display, people flexibly use table lines when designing tables, resulting in various cases of lines crossing cells, which limits the model's representational capabilities. Summary of the Invention

[0005] The purpose of this invention is to provide a table structure recognition method based on edge convolution interaction, so as to solve the problem that existing table structure recognition methods cannot accurately determine the relationship between table cells.

[0006] The technical solution adopted in this embodiment of the invention is: a table structure recognition method based on edge convolution interaction, comprising the following steps:

[0007] First, we construct a table structure recognition model based on edge convolutional interactions:

[0008] Step 1: Input the table image, the table feature map, the text row content information and position information in the table. The text row content information refers to the text content, and the text row position information refers to the coordinates of the text row. Based on the text row content information and position information, obtain the text center position, and sample the corresponding position on the feature map. Concatenate the sampled image features and the text row position information to obtain the image features containing the text position.

[0009] Step 2: Represent the table as a network graph, using the text rows in the table as vertices and the relationships between the text rows as edges;

[0010] Step 3: Calculate the edge features of each vertex and its M nearest vertices based on edge convolution, and then perform interactive aggregation operation on the vertex and the M edge features;

[0011] Step 4: Based on the feature vectors aggregated from the interactions of each vertex, a classification network is used to classify the relationships between vertices, and the relationship categories between vertices are the relationships between the text rows in the table.

[0012] Then, the table structure recognition model based on edge convolution interaction is trained;

[0013] Finally, the table image to be identified, the feature map of the table image, the text line content information and position information in the table image are input into the trained table structure recognition model based on edge convolution interaction to perform table structure recognition.

[0014] Furthermore, in step 2, the relationships between text lines are categorized into four types: no relationship, same cell, same row, and same column; the table is represented in the form of a network diagram using the following formula:

[0015]

[0016] Where V is the set of vertices in the network graph, E is the set of edges in the network graph, and the relationship between vertices is represented by an adjacency matrix of size V×V, where each element of the adjacency matrix represents the relationship category between the corresponding two vertices.

[0017] In step 4, based on the feature vectors aggregated from the interactions of each vertex, a classification network is used to classify the relationships between vertices. This involves calculating and updating the adjacency matrix. For each element of the adjacency matrix, the relationship category with the highest score probability is the predicted relationship category between the two vertices.

[0018] Furthermore, the specific implementation process of step 3 is as follows:

[0019] Step 31: Use graph embedding to map the image features containing text locations at each vertex in the network graph to Euclidean space:

[0020] Let the number of text lines, i.e., the number of vertices in the network graph, be n; the dimension of the sampled feature map be d; and the number of text line position information points be p. After concatenation, the feature of each vertex in the network graph, i.e., the dimension F of the image feature containing the text position, is the linear sum of the feature map dimension d and the number of text box position features p. The feature vector group of the vertices in the network graph is represented by the following formula:

[0021]

[0022] Among them, X i Let x represent the i-th point in Euclidean space. i eigenvectors;

[0023] Step 32: Calculate the edge features of each vertex and its M nearest neighbors based on edge convolution:

[0024] In Euclidean space, for each point x i Find M distance points x using the KNN algorithm i The nearest point x ij , take M points x ij Sort by distance in ascending order to get point x ij1 x ij2 , ..., x ijM ;

[0025] For point x i eigenvector X i Sum and point x i The distance between the M nearest points x ij eigenvector X j Perform edge convolution to obtain point x i and x ijm edge feature e ijm , m∈[1,M];

[0026] Step 33: Transfer edge features e ijm and point x i eigenvector X i The interactive aggregation process is shown in the following formula:

[0027]

[0028] in, Representing point x iThe feature vectors after edge convolution interaction, where ⊙ represents the Hadamard product, w ij It is a learnable vector, u i It is a learnable scalar;

[0029] Step 34: Using point x i A target point set is formed. From the vertex set V of the network graph, the point closest to any target point in the target point set is selected as a new target point. If the selected point has previously been a target point, it is discarded. Then, another point (excluding the selected point) closest to any target point in the target point set is selected as a new target point. The new target point is added to the target point set, and the process returns to step 32. Edge convolution interactions are performed on the new target point to obtain its feature vector after edge convolution interaction. This process is repeated until the vertex set V is traversed, outputting the feature vector after edge convolution interaction for each vertex. Input this as the feature vector after interaction aggregation of each vertex into step 4.

[0030] Furthermore, step 3 also includes:

[0031] Step 35: Convolve the feature vectors obtained from the edges of each vertex. The first fully connected layer is input, and it maps the learned feature representations to the label space of the samples and outputs them through the ReLU function.

[0032] Step 36: Use the new feature vector output from the previous step S35 as the feature vector X of the corresponding vertex in the next iteration. i Repeat steps 32-35 twice, then repeat steps 32-34 once more, to obtain the feature vectors after edge convolution interaction for each vertex. Then the feature vector Input the second fully connected layer, concatenate and combine them according to the following formula, and output them through the ReLU function:

[0033]

[0034] Where, X′ i Let x be the point i The feature vector W after multiple edge convolution interactions G The weight matrix is ​​a learnable matrix;

[0035] Point x i The feature vector X′ after multiple edge convolution interactions i Step 4 inputs the feature vectors after interaction aggregation for each vertex.

[0036] Furthermore, the edge convolution calculation in step 32 is shown in the following formula:

[0037] e ijm =ReLU(θ)m ·(X j -X i )+φ m ·X i );

[0038] Where, φ m and θ m X is a learnable vector; i =(k i1 , ..., k id x i1 y i1 x i2 y i2 ), (x i1 y i1 x i2 y i2 Let x be a point. i The corner coordinates are the coordinates of the i-th text line, where (x... i1 y i1 Let (x) be the coordinate of the top-left corner of the i-th text line. i2 y i2 Let k be the coordinate of the bottom right corner of the i-th text line. i1 Let x be the point i The first-dimensional eigenvector, k id Let x be the point i The d-th eigenvector; ReLU is the activation function.

[0039] Furthermore, n is the number of lines of text.

[0040] Furthermore, when training the table structure recognition model based on edge convolution interaction, before using a classification network to classify the relationships between vertices, the adjacency matrix is ​​downsampled using the Monte Carlo method, the number of each category in the labels of the adjacency matrix is ​​counted, the average number of categories is used to obtain the probability distribution matrix, the probability distribution map is sampled to generate the corresponding position index, and finally the feature map of the table is sliced ​​according to the position index, the adjacency matrix is ​​compressed, the number of categories is balanced, and batch training is completed.

[0041] Furthermore, when constructing a table structure recognition model based on edge convolutional interactions, the following are also included:

[0042] Step 5: Restore the table structure based on the relationship between the text rows in the table.

[0043] The beneficial effects of this embodiment are as follows: This embodiment extracts feature maps of table images from a backbone network composed of multi-layer convolutional neural networks, generates text center positions based on the text row content and position information in the table image, and samples the corresponding positions on the feature map to obtain image features containing text positions; the table is represented in the form of a network graph, where each vertex represents a text row; then, edge features of the M nearest points connected to the vertex are calculated based on edge convolution, and the edge features between the vertex and its M nearest connected points are interactively aggregated; finally, a multi-layer perceptron is used for classification and judgment to obtain the adjacency relationship of each text row in the table, more accurately capturing the relationship between each text row, and recovering the table structure based on the adjacency relationship between text rows, solving the problem that existing table structure recognition methods cannot accurately determine the relationship between table cells, and the recognized table structure is more accurate. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 This is a flowchart of a table structure recognition method based on edge convolution interaction according to an embodiment of the present invention.

[0046] Figure 2 The edge feature e is formed by the vertex-to-vertex edge operation in this embodiment of the invention. ijm A schematic diagram.

[0047] Figure 3 This is a flowchart of edge convolution interaction in an embodiment of the present invention. Detailed Implementation

[0048] The technical solutions of the present invention will be clearly and completely described below with reference to the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0049] Example 1

[0050] This embodiment provides a table structure recognition method based on edge convolution interaction, which specifically includes the following steps:

[0051] First, a table structure recognition model based on edge convolution interaction is constructed, and the specific steps are as follows:

[0052] Step 1: Input the table image, the table's feature map, and the text row content and position information in the table. A text row refers to a line of information for text detection and recognition. Each cell recognizes at least one text row. The text row content information refers to the text content of the text row, and the text row position information refers to the coordinates of the text row. Based on the text row content and position information, the text center position is obtained, and samples are taken at the corresponding position on the feature map to focus more on the features at the text row recognition location. The sampled image features and the text row position information are concatenated to obtain the image features containing the text position.

[0053] Step 2: Using the text rows in the table as vertices and the relationships between text rows as edges, represent the table in the form of a network diagram. The relationships between text rows are divided into four categories: no relationship, same cell, same row, and same column, as shown in the following formula:

[0054]

[0055] Where V is the set of vertices in the network graph, E is the set of edges in the network graph, and the relationship between vertices is represented by an adjacency matrix of size V×V, where each element of the adjacency matrix represents the relationship category between the corresponding two vertices.

[0056] Step 3: Calculate the edge features between each vertex and its M nearest vertices based on edge convolution. The edge convolution interaction process is as follows: Figure 3 As shown, the vertex and M edge features are then interactively aggregated. The specific implementation process is as follows:

[0057] Step 31: Use graph embedding to map the image features containing text locations of each vertex in the network graph to Euclidean space, aggregating local and neighborhood information of the location:

[0058] Let the number of text lines, i.e., the number of vertices in the network graph, be n; the dimension of the sampled feature map be d; and the number of text line position information points be p. After concatenation, the feature of each vertex in the network graph, i.e., the dimension F of the image feature containing the text position, is the linear sum of the feature map dimension d and the number of text box position features p. The feature vector group of the vertices in the network graph is represented by the following formula:

[0059]

[0060] Among them, X i Let x represent the i-th point in Euclidean space. i eigenvectors;

[0061] Step 32: Calculate the edge features of each vertex and its M nearest neighbors based on edge convolution:

[0062] In Euclidean space, for each point x i Find M distance points x using the KNN algorithm i The nearest point x ij ,in, M is determined by the number of text lines n, and M points x ij Sort by distance in ascending order to get point x ij1 x ij2 ,....,x ijM ;

[0063] For point x i eigenvector X i Sum and point x i The distance between the M nearest points x ij eigenvector X j Perform edge convolution operations, such as Figure 2 As shown, the edge convolution is calculated as follows:

[0064] e ijm =ReLU(θ) m ·(X j -X i )+φ m ·X i );

[0065] Among them, e ijm Let x be the point i and x ijm edge features, φ m and θ m Let X be a learnable parameter (vector), m∈[1,M]; i =(k i1 , ..., k id x i1 y i1 x i2 y i2 ), (x i1 y i1 x i2 y i2 Let x be a point. i The corner coordinates are the coordinates of the i-th text line, where (x... i1 y i1 Let (x) be the coordinate of the top-left corner of the i-th text line. i2 y i2 Let k be the coordinate of the bottom right corner of the i-th text line. i1 Let x be the point i The first-dimensional eigenvector, k id Let x be the point i The d-th eigenvector; ReLU is the activation function;

[0066] Step 33: Transfer edge features e ijm and point x i eigenvector X i To perform interactive aggregation, the operation process is as follows: Figure 3 As shown, the calculation process is as follows:

[0067]

[0068] in, Representing point x i The feature vectors after edge convolution interaction, where ⊙ represents the Hadamard product, w ij It is a learnable parameter (vector), u i It is a learnable scalar;

[0069] Step 34: Using point x i A target point set is formed. From the vertex set V of the network graph, the point closest to any target point in the target point set is selected as a new target point. If the selected point has previously been a target point, it is discarded. Then, another point (excluding the selected point) closest to any target point in the target point set is selected as a new target point. The new target point is added to the target point set, and the process returns to step 32. Edge convolution interactions are performed on the new target point to obtain its feature vector after edge convolution interaction. This process is repeated until the vertex set V is traversed, outputting the feature vector after edge convolution interaction for each vertex.

[0070] Step 35: Convolve the feature vectors obtained from the edges of each vertex. The first fully connected layer is input, and it maps the learned feature representations to the label space of the samples and outputs them through the ReLU function.

[0071] Step 36: Use the new feature vector output from the previous step S35 as the feature vector X of the corresponding vertex in the next iteration. i Repeat steps 32-35 three times, then repeat steps 32-34 once more, to obtain the feature vectors after edge convolution interaction for each vertex. That is, calculation Time X, as the feature vector in step 32 i ,calculate Time X, as the feature vector in step 32 i ,calculate Time X, as the feature vector in step 32 i Then the feature vector Input the second fully connected layer, concatenate and combine them according to the following formula, and output them through the ReLU function:

[0072]

[0073] Where, X′ i Let x be the point i The feature vector W after multiple edge convolution interactions G The weight matrix is ​​a learnable matrix;

[0074] Step 4: The feature vector X′ is generated based on multiple edge convolution interactions at each vertex. i The classification network is used to classify the relationships between vertices. Specifically, it calculates and updates the adjacency matrix that represents the relationship category between any two vertices in the network graph. The relationship category between the vertices is the relationship between each text line in the table. For each element of the adjacency matrix, the relationship category with the highest score probability is the predicted relationship category between the two vertices. The classification network consists of a batch normalization layer and a multilayer perceptron, which transforms the feature dimension.

[0075] Step 5: Restore the table structure based on the relationship between the text rows in the table.

[0076] Then, the table structure recognition model based on edge convolution interaction is trained:

[0077] During training, due to memory limitations, batch training is impossible if each position in the adjacency matrix of a text line is classified, i.e., each element value is calculated. Therefore, computational complexity must be reduced during training. Furthermore, the relationship categories of elements in the adjacency matrix of text lines are highly imbalanced, with the vast majority being unrelated. Therefore, when training a table structure recognition model based on edge convolutional interactions, before using a classification network to classify the relationships between vertices, the adjacency matrix is ​​downsampled using the Monte Carlo method. The number of each category in the labels of the adjacency matrix is ​​counted, and the average number of categories is used to obtain a probability distribution matrix. Matching position indices are generated by sampling from the probability distribution map. Finally, the feature map of the table is sliced ​​based on the position indices, the adjacency matrix is ​​compressed, the number of categories is balanced, and batch training is completed.

[0078] Finally, the table image to be identified, the feature map of the table image, the text line content information and position information in the table image are input into the trained table structure recognition model based on edge convolution interaction to perform table structure recognition.

[0079] This embodiment extracts feature maps from a table image using a backbone network composed of multi-layer convolutional neural networks. The text detection results (text row content and position information in the table image) generate the text center position, and samples are taken at the corresponding location on the feature map to obtain image features containing the text position. The table is represented as a network graph, where each vertex represents a text row. Then, edge features are calculated based on edge convolution to the M nearest points connected to the vertex. The edge features between the vertex and its M nearest connected points are then interactively aggregated. Finally, a multi-layer perceptron is used for classification to obtain the adjacency relationships of each text row in the table, more accurately capturing the relationships between each text row. The table structure is recovered based on the adjacency relationships between text rows, resulting in more accurate recognition results.

[0080] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A table structure recognition method based on edge convolution interaction, characterized in that, Includes the following steps: First, construct a table structure recognition model based on edge convolutional interactions: Step 1: Input the table image, the table feature map, the text row content information and position information in the table. The text row content information refers to the text content, and the text row position information refers to the coordinates of the text row. Based on the text row content information and position information, obtain the text center position, and sample the corresponding position on the feature map. Concatenate the sampled image features and the text row position information to obtain the image features containing the text position. Step 2: Represent the table as a network graph, using the text rows in the table as vertices and the relationships between the text rows as edges; Step 3: Calculate the edge features of each vertex and its M nearest vertices based on edge convolution, and then perform interactive aggregation operation on the vertex and the M edge features; Step 3 specifically includes: Step 31: Use graph embedding to map the image features containing text locations of each vertex in the network graph to Euclidean space; Step 32: Calculate the edge features of each vertex and its M nearest neighbors based on edge convolution; The edge convolution is calculated as follows: e ijm =ReLU(θ m ·(X j -X i )+ϕ m ·X i ): Among them, φ m and θ m X is a learnable vector; i =(k i1 ,…,k id ,x i1 ,y i1 ,x i2 ,y i2 ), (x i1 ,y i1 ,x i2 ,y i2 Let x be a point. i The corner coordinates are the coordinates of the i-th text line, where (x... i1 ,y i1 Let (x) be the coordinate of the top-left corner of the i-th text line. i2 ,y i2 Let k be the coordinate of the bottom right corner of the i-th text line. i1 Let x be the point i The first-dimensional eigenvector, k id Let x be the point i The d-th eigenvector; ReLU is the activation function; Step 33: Transfer edge features e ijm and point x i eigenvector X i The interactive aggregation process is shown in the following formula: in, Representing point x i The feature vectors after edge convolution interaction, where ⊙ represents the Hadamard product, w ij It is a learnable vector, u i It is a learnable scalar; Step 34: Using point x i A target point set is formed. From the vertex set V of the network graph, the point closest to any target point in the target point set is selected as a new target point. If the selected point has previously been a target point, it is discarded. Then, another point (excluding the selected point) closest to any target point in the target point set is selected as a new target point. The new target point is added to the target point set, and the process returns to step 32. Edge convolution interactions are performed on the new target point to obtain its feature vector after edge convolution interaction. This process is repeated until the vertex set V is traversed, outputting the feature vector after edge convolution interaction for each vertex. Input this as the feature vector after interaction aggregation of each vertex into step 4; Step 4: Based on the feature vectors aggregated from the interactions of each vertex, a classification network is used to classify the relationships between vertices, and the relationship categories between vertices are the relationships between the text rows in the table. Then, the table structure recognition model based on edge convolution interaction is trained; Finally, the table image to be identified, the feature map of the table image, the text line content information and position information in the table image are input into the trained table structure recognition model based on edge convolution interaction to perform table structure recognition.

2. The table structure recognition method based on edge convolution interaction according to claim 1, characterized in that, In step 2, the relationships between text lines are divided into four categories: no relationship, same cell, same row, and same column; The table can be represented as a network diagram using the following formula: Where V is the set of vertices in the network graph, E is the set of edges in the network graph, and the relationship between vertices is represented by an adjacency matrix of size V×V, where each element of the adjacency matrix represents the relationship category between the corresponding two vertices. In step 4, based on the feature vectors aggregated from the interactions of each vertex, a classification network is used to classify the relationships between vertices. This involves calculating and updating the adjacency matrix. For each element of the adjacency matrix, the relationship category with the highest score probability is the predicted relationship category between the two vertices.

3. The table structure recognition method based on edge convolution interaction according to claim 1, characterized in that, Step 31 includes: Let the number of text lines, i.e., the number of vertices in the network graph, be n; the dimension of the sampled feature map be d; and the number of text line position information points be p. After concatenation, the feature of each vertex in the network graph, i.e., the dimension F of the image feature containing the text position, is the linear sum of the feature map dimension d and the number of text box position features p. The feature vector group of the vertices in the network graph is represented by the following formula: Among them, X i Let x represent the i-th point in Euclidean space. i eigenvectors; Step 32 includes: In Euclidean space, for each point x i Find M distance points x using the KNN algorithm i The nearest point x ij , take M points x ij Sort by distance in ascending order to get point x ij1 ,x ij2 ,…,x ijM ; For point x i eigenvector X i Sum and point x i The distance between the M nearest points x ij eigenvector X j Perform edge convolution to obtain point x i and x ijm edge feature e ijm , m∈[1,M].

4. The table structure recognition method based on edge convolution interaction according to claim 3, characterized in that, Step 3 also includes: Step 35: Convolve the feature vectors obtained from the edges of each vertex. The first fully connected layer is input, and it maps the learned feature representations to the label space of the samples and outputs them through the ReLU function. Step 36: Use the new feature vector output from the previous step S35 as the feature vector X of the corresponding vertex in the next iteration. i Repeat steps 32-35 twice, then repeat steps 32-34 once more, to obtain the feature vectors after edge convolution interaction for each vertex. Then the feature vector Input the second fully connected layer, concatenate and combine them according to the following formula, and output them through the ReLU function: Among them, X i ′ Let x be the point i The feature vector W after multiple edge convolution interactions G The weight matrix is ​​a learnable matrix; Point x i The feature vector X after multiple edge convolution interactions i ′ Step 4 inputs the feature vectors after interaction aggregation for each vertex.

5. A table structure recognition method based on edge convolution interaction according to claim 3 or 4, characterized in that, n is the number of lines of text.

6. A table structure recognition method based on edge convolution interaction according to any one of claims 1 to 4, characterized in that, When training a table structure recognition model based on edge convolution interaction, before using a classification network to classify the relationships between vertices, the adjacency matrix is ​​downsampled using the Monte Carlo method. The number of each category in the labels of the adjacency matrix is ​​counted, and the average number of categories is used to obtain a probability distribution matrix. The probability distribution map is sampled to generate the corresponding position index. Finally, the feature map of the table is sliced ​​according to the position index, the adjacency matrix is ​​compressed, the number of categories is balanced, and batch training is completed.

7. A table structure recognition method based on edge convolution interaction according to any one of claims 1 to 4, characterized in that, When constructing a table structure recognition model based on edge convolutional interactions, the following are also included: Step 5: Restore the table structure based on the relationship between the text rows in the table.

Images