Table structure identification method and system based on multi-scale alignment
By constructing a feature pyramid and using a bidirectional LSTM network, a multi-scale aligned table structure recognition method is adopted, which solves the feature misalignment problem caused by the scale difference of table elements and improves the accuracy and generalization ability of table recognition.
Patent Information
- Application Number
- CN202410623263.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-05-20
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2044-05-20
AI Technical Summary
Existing table structure recognition methods lack generalization ability, fail to effectively consider feature misalignment caused by differences in table element size, affect contextual association, and have scale imbalance.
A multi-scale aligned table structure recognition method is adopted. By constructing a multi-scale aligned table structure recognition model, including row feature extraction, column feature extraction, feature pyramid construction and grid cell merging, feature mapping and merging judgment are performed using FPN network and bidirectional LSTM network to achieve standardization and quantification of table elements.
It effectively solves the problem of scale heterogeneity of table elements, improves the generalization ability and accuracy of table recognition, and realizes the context association and merging accuracy of table elements.
Smart Images

Figure CN118942103A_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the field of OCR technology, and in particular relates to a table structure recognition technology. Background Art
[0002] The most common elements in documents include text, charts, tables, formulas, etc. Among them, tables, as an important part of documents, organize content in a structured form, can convey more information with fewer words, and can quickly retrieve, compare and understand content.
[0003] However, since most tables are presented in non-editable forms (images, PDF), the processing and analysis of large amounts of table data is traditionally done manually, which is a time-consuming and error-prone task. With the trend of digital transformation and office electronicization, automatic table recognition has become an important topic in the field of document recognition and has attracted the attention of many scholars. In the process of table recognition, since the relational structure in the table determines the layout of the table, table structure recognition has become a key task that needs to be solved urgently.
[0004] The purpose of table structure recognition (TSR) is to reconstruct the cell structure of each detected table, that is, to identify the coordinates of each cell bounding box and its row and column arrangement information, and ultimately realize the conversion of the table from a non-editable file format (such as image, PDF) to an editable file format (such as text document, Excel table), effectively reducing labor costs while adapting to the needs of big data development.
[0005] However, due to the diversity of document types and table structures, the task of extracting the structure and content of a table is difficult.
[0006] In the early days, some scholars tried to identify table structures through heuristic and rule-based methods. These methods usually use optical features (such as text arrangement, vertical and horizontal dividing lines of tables, or templates) to detect tables in specific scenarios or with simple structures. This type of method requires the design of a large number of heuristic rules and hyperparameters. Although some scholars have proposed statistical and machine learning-based methods to parse tables in recent years, they still lack generalization capabilities.
[0007] In recent years, many deep learning-based methods have achieved excellent results in table recognition. However, the general approach of previous methods is to first identify rows and columns from the table to locate the table grid elements, and then determine the merging relationship between adjacent grid elements to establish logical associations. The grid elements here are the basic units that make up the table, and each cell is composed of one or more corresponding grid elements. However, the disadvantage of this method is that when mapping the features of the table elements, the size difference (scale difference) of the table elements is not taken into account, which causes the misalignment of the table element features, thereby affecting the contextual association of the table element features.
[0008] Previous solutions often use the ROI-align method to map table elements after obtaining the table row and column regions. However, mapping table elements of different sizes on the same scale feature map ignores the scale heterogeneity of different table elements, resulting in scale imbalance. Summary of the invention
[0009] The present invention proposes a table structure recognition method and system based on multi-scale alignment to solve the problems existing in the existing table structure recognition methods: 1. The generalization ability is insufficient; 2. When the table elements are mapped, the size difference (scale difference) of the table elements is not taken into account, which leads to the misalignment of the table element features, thereby affecting the context association of the table element features; 3. The practice of mapping table elements of different sizes on the same scale feature map ignores the scale heterogeneity of different table elements, resulting in the problem of scale imbalance.
[0010] The present invention proposes a table structure recognition method based on multi-scale alignment, comprising:
[0011] S1: Obtain the original table data, annotations and table images, and perform preprocessing to obtain the training set;
[0012] S2: Construct a multi-scale alignment table structure recognition model, including:
[0013] S2.1: Extract row features and column features from the table image, identify the row and column areas of the table, and split the table into a grid structure;
[0014] S2.2: constructing a row and column feature pyramid to obtain a feature map of each grid cell in the grid structure;
[0015] S2.3: Identify and restore merged cells;
[0016] S3: using the training set to train the multi-scale aligned table structure recognition model to obtain a trained multi-scale aligned table structure recognition model, and using the trained multi-scale aligned table structure recognition model to process a table to be recognized to obtain a table structure recognition result.
[0017] Furthermore, a preferred solution is provided: S1 comprises:
[0018] S1.1: Parse the table HTML structure and the coordinates of the marked cell bounding box to obtain the table information;
[0019] S1.2: Split each cell in the table into grid cells, and obtain the merging relationship between each adjacent grid cell;
[0020] S1.3: Transform the size of the table image into an input size, and linearize and normalize the voxel values of the table image.
[0021] Furthermore, a preferred solution is provided: the information of the table includes: the number of rows, the number of columns, the row area, and the column area of the table; the input size is (224, 224, 3); and the range of the linearization normalization is [0, 1].
[0022] Furthermore, a preferred solution is provided: S2.1 includes:
[0023] S2.1.1: Convolve the table image using a 1*3 convolution kernel and a 3*1 convolution kernel respectively, and use the pooling method to pool the convolved table image in the horizontal and vertical directions to obtain row features and column features;
[0024] S2.1.2: Use the classifier to predict the row features and column features, and determine whether the image region corresponding to each position on the vectors of the row features and column features is a row region and a column region;
[0025] S2.1.3: Intersect the table row and column regions to break the table into a grid structure.
[0026] Furthermore, a preferred solution is provided: S2.2 includes:
[0027] S2.2.1: Based on the construction method of the FPN network, a feature pyramid network is constructed for row features and column features to extract row features and column features from low to high;
[0028] S2.2.2: Map the obtained row features and column features to the corresponding row pyramid feature level and column feature pyramid level according to the size of each grid unit, and obtain the corresponding feature embedding;
[0029] S2.2.3: Combine the row-wise features and column-wise features of each grid cell to obtain a feature map for each grid cell.
[0030] Furthermore, a preferred solution is provided: S2.3 includes:
[0031] S2.3.1: Obtain the feature matrix of each grid unit and input it into the bidirectional LSTM network to construct the row and column context semantics between each grid unit;
[0032] S2.3.2: Use a classifier to determine whether the upper and lower adjacent grid cells and the left and right adjacent grid cells should be merged, and restore the merged cells.
[0033] Furthermore, a preferred solution is provided: the S3 includes: inputting the training set into the multi-scale alignment table structure recognition model, calculating the corresponding loss according to the loss function, and performing back propagation to update parameters.
[0034] The present invention also proposes a table structure recognition system based on multi-scale alignment, comprising:
[0035] Processing module: used to obtain original table data, annotations and table images, and perform preprocessing to obtain training sets;
[0036] Modeling module: used to build a multi-scale alignment table structure recognition model, including: extracting row features and column features from table images, identifying the row and column areas of the table, and splitting the table into a grid structure; building a row and column feature pyramid to obtain a feature map of each grid cell in the grid structure; identifying and restoring merged cells;
[0037] Recognition module: used to train the multi-scale alignment table structure recognition model using the training set to obtain the trained multi-scale alignment table structure recognition model, and use the trained multi-scale alignment table structure recognition model to process the table to be recognized to obtain a table structure recognition result.
[0038] The present invention also proposes a computer device, including a memory and a processor, wherein a computer program is stored in the memory. When the processor runs the computer program stored in the memory, the processor executes the table structure recognition method based on multi-scale alignment described in any combination of the above-mentioned schemes.
[0039] The present invention also proposes a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the steps of the table structure recognition method based on multi-scale alignment described in any combination of the above solutions.
[0040] Compared with the prior art, the advantages of the present invention are:
[0041] The present invention proposes a table structure recognition method PL-TSR based on multi-scale table element feature mapping and context information enhancement. By constructing a feature pyramid for mapping, table grid elements of various sizes are mapped to features of the same dimension, achieving standardization and measurability of table elements.
[0042] The present invention proposes a table element merging method that considers the logical correlation of rows and columns. The method uses the row and column neighborhood of the table element as the receptive field to enhance the characteristics of the table element. The method uses the logical context association of the table element as the basis for table merging judgment, which can effectively solve the problem of distinguishing the table structure where cells occupy multiple rows / columns.
[0043] The present invention is applicable to text search and table recognition scenarios. BRIEF DESCRIPTION OF THE DRAWINGS
[0044] Figure 1 This is a schematic diagram of an example of constructing a feature pyramid described in a specific embodiment of the present invention;
[0045] Figure 2 It is the overall framework diagram of GSSC described in the specific implementation mode of the present invention;
[0046] Figure 3 It is a schematic diagram of the merging decision module described in a specific implementation manner of the present invention. DETAILED DESCRIPTION
[0047] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0048] Implementation Method 1
[0049] Reference Figure 1 , Figure 2 , Figure 3 This embodiment will be described.
[0050] The table structure recognition method based on multi-scale alignment described in this embodiment includes:
[0051] S1: Obtain original table data, annotated data and table images, and perform preprocessing to obtain a training set;
[0052] S1.1: Parse the table HTML structure and the coordinates of the marked cell bounding box to obtain the table information, including: the number of rows, columns, row area, and column area of the table;
[0053] S1.2: Split each cell in the table into grid cells, and obtain the merging relationship between each adjacent grid cell;
[0054] S1.3: Transform the size of the table image to the input size (224, 224, 3), and
[0055] The prime values are linearly normalized to the range [0,1];
[0056] S2: Construct a multi-scale aligned table structure recognition model (PL-TSR model), including:
[0057] S2.1: Extract row features and column features from the table image, identify the row and column areas of the table, and split the table into a grid structure;
[0058] S2.1.1: Convolve the table image using a 1*3 convolution kernel and a 3*1 convolution kernel respectively, and use the pooling method to pool the convolved table image in the horizontal and vertical directions to obtain row features and column features;
[0059] S2.1.2: Use two classifiers based on fully connected layers to predict row features and column features, and determine whether the image region corresponding to each position on the vectors of row features and column features is a row region or a column region;
[0060] S2.1.3: Intersect the table row and column regions to split the table into a grid structure;
[0061] S2.2: constructing a row and column feature pyramid to obtain a feature map of each grid cell in the grid structure;
[0062] S2.2.1: Based on the construction method of the FPN network, a feature pyramid network is constructed for row features and column features to extract row features and column features from low to high;
[0063] S2.2.2: Map the obtained row features and column features to the corresponding row pyramid feature level and column feature pyramid level according to the size of each grid unit, and obtain the corresponding feature embedding;
[0064] S2.2.3: Combine the row-wise features and column-wise features of each grid cell to obtain a feature map of each grid cell;
[0065] S2.3: Identify and restore merged cells;
[0066] S2.3.1: Obtain the feature matrix of each grid unit and input it into the bidirectional LSTM network to construct the row and column context semantics between each grid unit;
[0067] S2.3.2: Use two fully connected network-based classifiers to determine whether to merge adjacent grid cells above and below and adjacent grid cells below and below, and restore the merged cells;
[0068] S3: Using the training set to train the multi-scale alignment table structure recognition model to obtain the trained multi-scale alignment table structure recognition model, including: inputting the training set into the multi-scale alignment table structure recognition model, calculating the corresponding loss according to the loss function, and performing back propagation to update the parameters; using the trained multi-scale alignment table structure recognition model to process the table to be recognized to obtain a table structure recognition result.
[0069] Implementation Method 2
[0070] This implementation is a further example of the table structure recognition method based on multi-scale alignment described in the first implementation.
[0071] In S2, in order to solve the situation where there are significant size differences in table cells, this embodiment uses a table cell feature mapping method based on multi-scale alignment. This method can adaptively map grid cells of different sizes to corresponding scales to obtain features of the corresponding scales. The scale-aligned grid cell features are then used in the subsequent merge cell judgment process.
[0072] The construction process of the feature pyramid is as follows:
[0073]
[0074] Among them, U_(×u) represents the upsampling operation, u is the upsampling ratio, conv represents the convolution operation, l i-1 represents the feature vector of the i-1th layer of the upsampling module sampling module, f i Represents the feature vector of the i-th layer of the downsampling module.
[0075] This results in a 7-layer feature pyramid output with sizes from low to high of [64, 32, 16, 8, 4, 2, 1].
[0076] The mapping function relationship between the pyramid level and the grid unit size and the spatial position mapping relationship are as follows:
[0077]
[0078]
[0079]
[0080]
[0081] in, Indicates e i The corresponding row and column feature pyramid levels after mapping. Indicates e i The feature position after mapping at the corresponding row and column pyramid level, M represents the height of the input feature map, h i Represents the grid unit e i The height of N represents the width of the input feature map, w i Represents the grid unit e i The width, x i Represents the grid unit e i The vertical coordinate, y i Represents the grid unit e i The horizontal coordinate of .
[0082] Furthermore, in S3, the trained PL-TSR model is used, and the loss function of the model mainly consists of two aspects, namely the loss of row and column region recognition and the loss of the grid unit merging process.
[0083] L=λ 1 L s +λ 2 L m
[0084] Among them, L s , L m Represent the row and column mask prediction loss and the table element merging loss, and L s , L m Both use binary cross entropy loss, λ 1 , 2 Respectively represent L s Weight and L m Weight.
[0085]
[0086] Among them, n represents the number of samples and p represents the classification probability.
[0087] Implementation Method 3
[0088] The table structure recognition system based on multi-scale alignment described in this embodiment includes:
[0089] Processing module: used to obtain original table data, annotations and table images, and perform preprocessing to obtain training sets;
[0090] Modeling module: used to build a multi-scale alignment table structure recognition model, including: extracting row features and column features from table images, identifying the row and column areas of the table, and splitting the table into a grid structure; building a row and column feature pyramid to obtain a feature map of each grid cell in the grid structure; identifying and restoring merged cells;
[0091] Recognition module: used to train the multi-scale alignment table structure recognition model using the training set to obtain the trained multi-scale alignment table structure recognition model, and use the trained multi-scale alignment table structure recognition model to process the table to be recognized to obtain a table structure recognition result.
[0092] Those skilled in the art will appreciate that the above are only preferred embodiments of the present invention, and the various embodiments of the present disclosure and / or the features described in the claims may be combined or combined in various ways, even if such combinations or combinations are not explicitly described in the present disclosure. It is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art may still modify the technical solutions described in the aforementioned embodiments, or perform equivalent substitutions on some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention shall be included in the protection scope of the present invention.
[0093] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.
[0094] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.
[0095] Those skilled in the art will appreciate that the embodiments of the present disclosure may be provided as methods, systems or computer program products. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0096] The present disclosure is described with reference to flowcharts and / or block diagrams of methods, devices (systems) and computer program products according to embodiments of the present disclosure. It should be understood that each process and / or block in the flowchart and / or block diagram and the combination of processes and / or blocks in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the functions specified in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0097] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0098] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present disclosure rather than to limit its protection scope. Although the present disclosure has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that after reading the present disclosure, those skilled in the art can still make various changes, modifications or equivalent substitutions to the specific implementation methods of the invention, but these changes, modifications or equivalent substitutions are all within the protection scope of the disclosed claims to be approved.
Claims
1. A table structure recognition method based on multi-scale alignment, characterized in that: The method comprises: S1: Obtain original table data, annotations and table images, and perform preprocessing to obtain a training set; S2: Construct a multi-scale alignment table structure recognition model, including: S2.1: Extract row features and column features from the table image, identify the row and column areas of the table, and split the table into a grid structure; S2.2: constructing a row and column feature pyramid to obtain a feature map of each grid cell in the grid structure; S2.3: Identify and restore merged cells; S3: using the training set to train the multi-scale aligned table structure recognition model to obtain a trained multi-scale aligned table structure recognition model, and using the trained multi-scale aligned table structure recognition model to process a table to be recognized to obtain a table structure recognition result.
2. The table structure recognition method based on multi-scale alignment according to claim 1, characterized in that: The S1 includes: S1.1: Parse the table HTML structure and the coordinates of the marked cell bounding box to obtain the table information; S1.2: Split each cell in the table into grid cells, and obtain the merging relationship between each adjacent grid cell; S1.3: Transform the size of the table image into an input size, and linearize and normalize the voxel values of the table image.
3. The table structure recognition method based on multi-scale alignment according to claim 2 is characterized in that: The information of the table includes: the number of rows, the number of columns, the row area, and the column area of the table; the input size is (224, 224, 3); and the range of the linearization normalization is [0, 1].
4. The table structure recognition method based on multi-scale alignment according to claim 1, characterized in that: The S2.1 includes: S2.1.1: Convolve the table image using a 1*3 convolution kernel and a 3*1 convolution kernel respectively, and use the pooling method to pool the convolved table image in the horizontal and vertical directions to obtain row features and column features; S2.1.2: Use the classifier to predict the row features and column features, and determine whether the image region corresponding to each position on the vectors of the row features and column features is a row region and a column region; S2.1.3: Intersect the table row and column regions to break the table into a grid structure.
5. The table structure recognition method based on multi-scale alignment according to claim 1, characterized in that: The S2.2 includes: S2.2.1: Based on the construction method of the FPN network, a feature pyramid network is constructed for row features and column features to extract row features and column features from low to high; S2.2.2: Map the obtained row features and column features to the corresponding row pyramid feature level and column feature pyramid level according to the size of each grid unit, and obtain the corresponding feature embedding; S2.2.3: Combine the row-wise features and column-wise features of each grid cell to obtain a feature map for each grid cell.
6. The table structure recognition method based on multi-scale alignment according to claim 1, characterized in that: The S2.3 includes: S2.3.1: Obtain the feature matrix of each grid unit and input it into the bidirectional LSTM network to construct the row and column context semantics between each grid unit; S2.3.2: Use a classifier to determine whether the upper and lower adjacent grid cells and the left and right adjacent grid cells should be merged, and restore the merged cells.
7. The table structure recognition method based on multi-scale alignment according to claim 1, characterized in that: The S3 includes: inputting the training set into the multi-scale alignment table structure recognition model, calculating the corresponding loss according to the loss function, and performing back propagation to update parameters.
8. A table structure recognition system based on multi-scale alignment, characterized in that: The system comprises: Processing module: used to obtain original table data, annotations and table images, and perform preprocessing to obtain training sets; Modeling module: used to build a multi-scale alignment table structure recognition model, including: extracting row features and column features from table images, identifying the row and column areas of the table, and splitting the table into a grid structure; building a row and column feature pyramid to obtain a feature map of each grid cell in the grid structure; identifying and restoring merged cells; Recognition module: used to train the multi-scale alignment table structure recognition model using the training set to obtain the trained multi-scale alignment table structure recognition model, and use the trained multi-scale alignment table structure recognition model to process the table to be recognized to obtain a table structure recognition result.
9. A computer device comprising a memory and a processor, characterized in that: A computer program is stored in the memory. When the processor runs the computer program stored in the memory, the processor executes the method according to any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.
Citation Information
Patent Citations
Table recognition method and device, electronic equipment and storage medium
CN115620321A
Table structure recognition method and training method and device of table structure recognition model
CN116259064A
Table analysis method, apparatus and device, and storage medium
CN116453146A
Table identification method, table identification model training method, apparatus and device
CN116645685A
Table image cell region identification method and system and medium
CN116778512A
Cited By
Archive management system and method based on artificial intelligence
CN121278159A
