Table structure recognition method and system based on multi-scale alignment

Abstract

The method and system for table structure recognition based on multi-scale alignment belong to the technical field of OCR, and solve the problems of the existing table structure recognition methods, such as lacking of generalization ability. The method comprises the following steps: obtaining original table data, labeled data and table images, and preprocessing to obtain a training set; constructing a multi-scale alignment table structure recognition model; training the multi-scale alignment table structure recognition model using the training set to obtain a trained multi-scale alignment table structure recognition model; and processing a table to be recognized using the trained multi-scale alignment table structure recognition model to obtain a table structure recognition result. The present application is suitable for text search and table recognition scenarios.

Description

Technical Field

[0001] The present invention belongs to the field of OCR technology, and in particular relates to a table structure recognition technology. Background Art

[0002] The most common elements in documents include text, charts, tables, formulas, etc. Among them, tables, as an important part of documents, organize content in a structured form, can convey more information with fewer words, and can quickly retrieve, compare and understand content.

[0003] However, because most tables are presented in non-editable formats (images, PDFs), processing and analyzing large amounts of tabular data has traditionally been performed manually, a time-consuming and error-prone task. With the trend toward digital transformation and the electronicization of office work, automated table recognition has become a key topic in the field of document recognition, attracting the attention of numerous scholars. In the table recognition process, because the relational structure within the table determines its layout, identifying the table structure has become a critical task that needs to be addressed.

[0004] The purpose of table structure recognition (TSR) is to reconstruct the cell structure of each detected table, that is, to identify the coordinates of each cell's bounding box and its row and column arrangement information, and ultimately realize the conversion of the table from an uneditable file format (such as images, PDFs) to an editable file format (such as text documents, Excel tables), effectively reducing labor costs while adapting to the needs of big data development.

[0005] However, due to the diversity of document types and table structures, the task of extracting the structure and content of a table is difficult.

[0006] Early researchers attempted to identify table structures using heuristic and rule-based methods. These methods typically leverage optical features (such as text alignment, vertical and horizontal table dividers, or templates) to detect tables in specific scenarios or with relatively simple structures. These methods require the design of numerous heuristic rules and hyperparameters. While recent researchers have proposed statistical and machine learning-based methods for parsing tables, these methods still lack generalization capabilities.

[0007] In recent years, a number of deep learning-based methods have achieved excellent results in table recognition. However, previous methods generally first identify rows and columns in the table to locate the table grid elements, and then determine the merging relationship between adjacent grid elements to establish logical associations. Here, grid elements are the basic units that make up the table; each cell is composed of one or more corresponding grid elements. However, the disadvantage of such methods is that they fail to consider the size differences (scale differences) of table elements when mapping features for table elements, resulting in misalignment of table element features, which in turn affects the contextual association of table element features.

[0008] Previous solutions often use ROI-align to map table elements after obtaining the row and column regions. However, mapping table elements of different sizes onto the same scale feature map ignores the scale heterogeneity of different table elements, resulting in scale imbalance. Summary of the Invention

[0009] The present invention proposes a table structure recognition method and system based on multi-scale alignment to solve the following problems existing in existing table structure recognition methods: 1. Lack of generalization ability; 2. Failing to consider size differences (scale differences) of table elements when performing feature mapping on table elements, resulting in misalignment of table element features, which in turn affects the contextual association of table element features; 3. Mapping table elements of different sizes on the same scale feature map ignores the scale heterogeneity of different table elements, resulting in scale imbalance.

[0010] The present invention proposes a table structure recognition method based on multi-scale alignment, comprising:

[0011] S1: Obtain the original table data, annotations and table images, and perform preprocessing to obtain the training set;

[0012] S2: Construct a multi-scale alignment table structure recognition model, including:

[0013] S2.1: Extract row and column features from the table image, identify the row and column areas of the table, and split the table into a grid structure;

[0014] S2.2: Constructing a row and column feature pyramid to obtain a feature map for each grid cell in the grid structure;

[0015] S2.3: Identify and restore merged cells;

[0016] S3: Using the training set to train the multi-scale aligned table structure recognition model to obtain a trained multi-scale aligned table structure recognition model, and using the trained multi-scale aligned table structure recognition model to process a table to be recognized to obtain a table structure recognition result.

[0017] Furthermore, a preferred solution is provided: S1 includes:

[0018] S1.1: Parse the table HTML structure and the coordinates of the marked cell bounding box to obtain the table information;

[0019] S1.2: Split each cell in the table into grid cells and obtain the merging relationship between each adjacent grid cell;

[0020] S1.3: Transform the size of the table image to the input size, and linearize and normalize the voxel values ​​of the table image.

[0021] Furthermore, a preferred solution is provided: the information of the table includes: the number of rows, the number of columns, the row area, and the column area of ​​the table; the input size is (224, 224, 3); and the range of the linearization normalization is [0, 1].

[0022] Furthermore, a preferred solution is provided: S2.1 includes:

[0023] S2.1.1: Convolve the table image using a 1*3 convolution kernel and a 3*1 convolution kernel respectively, and use the pooling method to pool the convolved table image in the horizontal and vertical directions to obtain row features and column features;

[0024] S2.1.2: Use the classifier to predict the row and column features and determine whether the image region corresponding to each position on the row and column feature vectors is a row region or a column region;

[0025] S2.1.3: Intersect the table row and column regions to break the table into a grid structure.

[0026] Furthermore, a preferred solution is provided: S2.2 includes:

[0027] S2.2.1: Based on the construction method of the FPN network, a feature pyramid network is constructed for row features and column features to extract row features and column features from low to high;

[0028] S2.2.2: Map the obtained row features and column features to the corresponding row pyramid feature level and column feature pyramid level according to the size of each grid cell, and obtain the corresponding feature embedding;

[0029] S2.2.3: Combine the row-wise features and column-wise features of each grid cell to obtain a feature map for each grid cell.

[0030] Furthermore, a preferred solution is provided: S2.3 includes:

[0031] S2.3.1: Obtain the feature matrix of each grid cell and input it into the bidirectional LSTM network to construct the row and column context semantics between each grid cell;

[0032] S2.3.2: Use a classifier to determine whether the upper and lower adjacent grid cells and the left and right adjacent grid cells should be merged, and restore the merged cells.

[0033] Furthermore, a preferred solution is provided: the S3 includes: inputting the training set into the multi-scale alignment table structure recognition model, calculating the corresponding loss according to the loss function, and performing back propagation to update parameters.

[0034] The present invention also proposes a table structure recognition system based on multi-scale alignment, comprising:

[0035] Processing module: used to obtain original table data, annotations and table images, and perform preprocessing to obtain training sets;

[0036] Modeling module: used to build a multi-scale alignment table structure recognition model, including: extracting row and column features from table images, identifying the row and column regions of the table, and splitting the table into a grid structure; constructing a row and column feature pyramid to obtain the feature map of each grid cell in the grid structure; and identifying and restoring merged cells;

[0037] Recognition module: used to use the training set to train the multi-scale aligned table structure recognition model to obtain the trained multi-scale aligned table structure recognition model, use the trained multi-scale aligned table structure recognition model to process the table to be recognized, and obtain a table structure recognition result.

[0038] The present invention also proposes a computer device comprising a memory and a processor, wherein a computer program is stored in the memory. When the processor runs the computer program stored in the memory, the processor executes the table structure recognition method based on multi-scale alignment described in any combination of the above-mentioned schemes.

[0039] The present invention also proposes a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the steps of the table structure recognition method based on multi-scale alignment described in any combination of the above solutions.

[0040] Compared with the prior art, the advantages of the present invention are:

[0041] The application provides a table structure recognition method PL-TSR based on multi-scale table element feature mapping and context information enhancement.

[0042] The application provides a table element merging method considering row-column logical correlation.

[0043] The application is suitable for text search and table recognition scenarios. DETAILED DESCRIPTION

[0044] Figure 1 The application provides a feature pyramid construction example schematic diagram described in the specific embodiment of the application;

[0045] Figure 2 The application provides a GSSC overall framework diagram described in the specific embodiment of the application;

[0046] Figure 3 The application provides a merging decision module schematic diagram described in the specific embodiment of the application. DETAILED DESCRIPTION

[0047] The technical solutions in the embodiments of the application will be clearly and completely described below with reference to the drawings of the application. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of the application.

[0048] Embodiment one

[0049] Reference Figure 1 , Figure 2 , Figure 3 The embodiment is described.

[0050] The table structure recognition method based on multi-scale alignment described in the embodiment includes:

[0051] S1: obtaining original table data, labeled data and table images, and performing preprocessing to obtain a training set;

[0052] S1.1: analyzing table html structure and labeled cell bounding box coordinates to obtain table information, including: number of rows, number of columns, row area and column area of the table;

[0053] S1.2: Split each cell in the table into grid cells and obtain the merging relationship between each adjacent grid cell;

[0054] S1.3: Convert the size of the table image to the input size (224, 224, 3), and

[0055] The prime values ​​are linearly normalized to the range of [0,1];

[0056] S2: Construct a multi-scale aligned table structure recognition model (PL-TSR model), including:

[0057] S2.1: Extract row and column features from the table image, identify the row and column areas of the table, and split the table into a grid structure;

[0058] S2.1.1: Convolve the table image using a 1*3 convolution kernel and a 3*1 convolution kernel respectively, and use the pooling method to pool the convolved table image in the horizontal and vertical directions to obtain row features and column features;

[0059] S2.1.2: Use two fully connected layer-based classifiers to predict row and column features, and determine whether the image region corresponding to each position on the row and column feature vectors is a row region or a column region;

[0060] S2.1.3: Intersect the table row and column regions to break the table into a grid structure;

[0061] S2.2: Constructing a row and column feature pyramid to obtain a feature map for each grid cell in the grid structure;

[0062] S2.2.1: Based on the construction method of the FPN network, a feature pyramid network is constructed for row features and column features to extract row features and column features from low to high;

[0063] S2.2.2: Map the obtained row features and column features to the corresponding row pyramid feature level and column feature pyramid level according to the size of each grid cell, and obtain the corresponding feature embedding;

[0064] S2.2.3: Combine the row-wise features and column-wise features of each grid cell to obtain a feature map for each grid cell.

[0065] S2.3: Identify and restore merged cells;

[0066] S2.3.1: Obtain the feature matrix of each grid cell and input it into the bidirectional LSTM network to construct the row and column context semantics between each grid cell;

[0067] S2.3.2: Use two fully connected network-based classifiers to determine whether to merge adjacent grid cells, and restore the merged cells.

[0068] S3: Using the training set to train the multi-scale aligned table structure recognition model to obtain a trained multi-scale aligned table structure recognition model, including: inputting the training set into the multi-scale aligned table structure recognition model, calculating the corresponding loss according to the loss function, and performing back propagation to update the parameters; using the trained multi-scale aligned table structure recognition model to process the table to be recognized to obtain a table structure recognition result.

[0069] Implementation Method 2

[0070] This embodiment is a further example of the table structure recognition method based on multi-scale alignment described in the first embodiment.

[0071] In step S2, to address the situation where table cells have significant size differences, this embodiment uses a table cell feature mapping method based on multi-scale alignment. This method can adaptively map grid cells of different sizes to corresponding scales, obtaining features at the corresponding scales. The scale-aligned grid cell features are then used in the subsequent merging cell determination process.

[0072] The process of constructing the feature pyramid is as follows:

[0073]

[0074] Among them, U_(×u) represents the upsampling operation, u is the upsampling ratio, conv represents the convolution operation, l i-1 Represents the feature vector of the i-1th layer of the upsampling module sampling module, f i Represents the feature vector of the i-th layer of the downsampling module.

[0075] This results in a 7-layer feature pyramid output with sizes from low to high of [64, 32, 16, 8, 4, 2, 1].

[0076] The mapping function relationship between pyramid levels and grid unit sizes, as well as the spatial position mapping relationship, are as follows:

[0077]

[0078]

[0079]

[0080]

[0081] in, Indicates e i The corresponding row and column feature pyramid levels after mapping. Indicates e i The feature position after mapping on the corresponding row and column pyramid level, M represents the height of the input feature map, h i Represents the grid unit e i The height of N represents the width of the input feature map, w i Represents the grid unit e i The width, x i Represents the grid unit e i The vertical coordinate, y i Represents the grid unit e i The horizontal coordinate of .

[0082] Furthermore, in S3, the trained PL-TSR model is used, and the loss function of the model mainly consists of two aspects, namely the loss of row and column region recognition and the loss of the grid unit merging process.

[0083] L=λ1L s +λ2L m

[0084] Among them, L s , L m Represent the row and column mask prediction loss and the table element merging loss, and L s , L m Both use binary cross entropy loss, λ1 and λ2 represent L s Weight and L m Weight.

[0085]

[0086] Among them, n represents the number of samples and p represents the classification probability.

[0087] Implementation Method 3

[0088] The table structure recognition system based on multi-scale alignment described in this embodiment includes:

[0089] Processing module: used to obtain original table data, annotations and table images, and perform preprocessing to obtain training sets;

[0090] Modeling module: used to build a multi-scale alignment table structure recognition model, including: extracting row and column features from table images, identifying the row and column regions of the table, and splitting the table into a grid structure; constructing a row and column feature pyramid to obtain the feature map of each grid cell in the grid structure; and identifying and restoring merged cells;

[0091] Recognition module: used to use the training set to train the multi-scale aligned table structure recognition model to obtain the trained multi-scale aligned table structure recognition model, use the trained multi-scale aligned table structure recognition model to process the table to be recognized, and obtain a table structure recognition result.

[0092] Those skilled in the art will understand that the above description is only a preferred embodiment of the present invention, and the features described in the various embodiments and / or claims of the present disclosure may be combined or coupled in various ways, even if such a combination or coupling is not explicitly described in the present disclosure. It is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art may still modify the technical solutions described in the aforementioned embodiments or make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention shall be included in the scope of protection of the present invention.

[0093] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0094] Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if such changes and modifications fall within the scope of the claims and their equivalents, the present invention is intended to include such changes and modifications.

[0095] Those skilled in the art will appreciate that embodiments of the present disclosure may be provided as methods, systems, or computer program products. Thus, the present disclosure may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0096] The present disclosure is described with reference to flowcharts and / or block diagrams of methods, devices (systems) and computer program products according to embodiments of the present disclosure. It should be understood that each process and / or block in the flowchart and / or block diagram and the combination of processes and / or blocks in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including the instruction device, which implements the function specified in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0097] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The steps for the function specified in one or more boxes.

[0098] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present disclosure and are not intended to limit its scope of protection. Although the present disclosure has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that after reading the present disclosure, those skilled in the art can still make various changes, modifications or equivalent substitutions to the specific implementation methods of the invention, but these changes, modifications or equivalent substitutions are all within the scope of protection of the disclosed claims.

Claims

1. A table structure recognition method based on multi-scale alignment, characterized by: The method comprises: S1: Obtain the original table data, annotations and table images, and perform preprocessing to obtain the training set; S2: Construct a multi-scale alignment table structure recognition model, including: S2.1: Extract row and column features from the table image, identify the row and column areas of the table, and split the table into a grid structure; S2.2: Constructing a row and column feature pyramid to obtain a feature map for each grid cell in the grid structure; S2.3: Identify and restore merged cells; S3: using the training set to train the multi-scale aligned table structure recognition model to obtain a trained multi-scale aligned table structure recognition model, and using the trained multi-scale aligned table structure recognition model to process the table to be recognized to obtain a table structure recognition result; Said S2.2 includes: S2.2.1: Based on the construction method of the FPN network, a feature pyramid network is constructed for row features and column features to extract row features and column features from low to high; S2.2.2: Map the obtained row features and column features to the corresponding row pyramid feature level and column feature pyramid level according to the size of each grid cell, and obtain the corresponding feature embedding; S2.2.3: Combine the row-wise features and column-wise features of each grid cell to obtain a feature map for each grid cell.

2. The table structure recognition method based on multi-scale alignment according to claim 1, characterized in that: Said S1 comprises: S1.1: Parse the table HTML structure and the coordinates of the marked cell bounding box to obtain the table information; S1.2: Split each cell in the table into grid cells and obtain the merging relationship between each adjacent grid cell; S1.3: Transform the size of the table image to the input size, and linearize and normalize the voxel values ​​of the table image.

3. The table structure recognition method based on multi-scale alignment according to claim 2, characterized in that: The table information includes: the number of rows, the number of columns, the row area, and the column area of ​​the table; the input size is (224, 224, 3); and the range of the linearization normalization is [0, 1].

4. The table structure recognition method based on multi-scale alignment according to claim 1, characterized in that: The S2.1 includes: S2.1.1: Convolve the table image using a 1*3 convolution kernel and a 3*1 convolution kernel, respectively. Pool the convolved table image horizontally and vertically using the pooling method to obtain row features and column features. S2.1.2: Use the classifier to predict the row and column features and determine whether the image region corresponding to each position on the row and column feature vectors is a row region or a column region; S2.1.3: Intersect the table row and column regions to break the table into a grid structure.

5. The table structure recognition method based on multi-scale alignment according to claim 1, characterized in that: Said S2.3 includes: S2.3.1: Obtain the feature matrix of each grid cell and input it into the bidirectional LSTM network to construct the row and column context semantics between each grid cell; S2.3.2: Use a classifier to determine whether the upper and lower adjacent grid cells and the left and right adjacent grid cells should be merged, and restore the merged cells.

6. The table structure recognition method based on multi-scale alignment according to claim 1, characterized in that: The S3 includes: inputting the training set into the multi-scale alignment table structure recognition model, calculating the corresponding loss according to the loss function, and performing back propagation to update parameters.

7. A table structure recognition system based on multi-scale alignment, characterized by: The system comprises: Processing module: used to obtain original table data, annotations and table images, and perform preprocessing to obtain training sets; Modeling module: used to build a multi-scale aligned table structure recognition model, including: extracting row and column features from the table image, identifying the row and column areas of the table, and splitting the table into a grid structure; constructing a row and column feature pyramid to obtain a feature map of each grid cell in the grid structure; constructing a feature pyramid network for row features and column features based on the FPN network construction method, extracting row features and column features from low to high; mapping the obtained row features and column features to the corresponding row pyramid feature level and column feature pyramid level according to the size of each grid cell, and obtaining the corresponding feature embedding; combining the row direction features and column direction features of each grid cell to obtain the feature map of each grid cell; identifying and restoring merged cells; Recognition module: used to use the training set to train the multi-scale aligned table structure recognition model to obtain the trained multi-scale aligned table structure recognition model, use the trained multi-scale aligned table structure recognition model to process the table to be recognized, and obtain a table structure recognition result.

8. A computer device comprising a memory and a processor, characterized in that: A computer program is stored in the memory. When the processor runs the computer program stored in the memory, the processor executes the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images