Table structure identification method, device and computer readable storage medium

Abstract

The application provides a table structure identification method and device and a computer readable storage medium. A table structure identification model capable of reasoning blank cell coordinates is trained, a table to be identified is obtained, and the table to be identified is input into the table structure identification model. The model automatically completes reasoning, high-precision, integrity and consistency identification of the wireless table structure of the table to be identified is achieved, and core technical problems such as table structure fracture and row and column misjudgment caused by missing blank cell coordinates in the prior art are effectively solved.

Description

Table structure recognition methods, devices and computer-readable storage media Technical Field

[0001] This application relates to the field of table recognition technology, and in particular to a table structure recognition method, device and computer-readable storage medium. Background Technology

[0002] With the significant success of Large Language Models (LLMs) in various natural language processing tasks, obtaining structured pre-training and fine-tuning data from high-quality documents such as academic literature, technical reports, and financial statements has become crucial. In these documents, tables, as the core carriers of high-density information, directly determine the quality of the upstream data through their recognition accuracy. According to ACM 2023 statistics, over 37% of quantitative results in STEM papers are presented in tables; while in industrial knowledge bases, this proportion exceeds 60%, covering key data such as equipment parameters and financial indicators.

[0003] Existing table recognition technologies are relatively mature when dealing with bordered tables. Their typical approach relies on detecting explicit visual cues: first, identifying table lines, then calculating intersections to form a grid, thereby resolving the cell structure. However, in practical applications such as academic journals, technical white papers, annual reports, and contracts, borderless tables are increasingly prevalent, yet their recognition accuracy is far lower than that of bordered tables. According to the public evaluation results of the ICDAR 2023 table recognition competition, the F1 score for structure resolution of borderless tables was only 68.4%, a significant gap of nearly 29 percentage points compared to bordered tables.

[0004] The fundamental challenge of wireless table structure recognition lies in the lack of visual cues. They usually only implicitly represent the structure through the alignment of text content (such as spaces and tabs). Once the document is slightly tilted or there is a character-level positioning error in OCR, the alignment of the entire row and column is easily disrupted.

[0005] Among these challenges, the recognition of blank cells presents a particularly prominent issue, constituting a significant technical bottleneck. Summary of the Invention

[0006] To overcome the problems existing in related technologies, this specification provides a method, device and computer-readable storage medium for table structure recognition.

[0007] Firstly, a table structure recognition method is provided, the method comprising:

[0008] Obtain the table to be recognized;

[0009] The table to be identified is input into a preset table structure recognition model, and the table structure information of the table to be identified is output.

[0010] The table structure recognition model is trained using a training dataset containing blank cells. The trained table structure recognition model can identify blank cells in the table to be recognized and output complete table structure information containing blank cells.

[0011] According to the table structure recognition method provided in this application, the table structure recognition model is trained using a training dataset containing blank cells, and includes:

[0012] Retrieve the original dataset corresponding to the original table containing both blank and non-blank cells;

[0013] Based on the topological structure information of the original table and the bounding box coordinates of the non-blank cells in the original table, the bounding box coordinates of the blank cells in the original table are completed, and an augmented dataset including the bounding box coordinates of the completed blank cells is generated.

[0014] The pre-trained table structure recognition model is fine-tuned using a training dataset formed by the original dataset and the augmented dataset.

[0015] According to the table structure recognition method provided in this application, the step of completing the bounding box coordinates of blank cells in the original table based on the topological structure information of the original table and the bounding box coordinates of non-blank cells in the original table includes:

[0016] A row and column boundary matrix is ​​established based on the topological structure information of the original table and the bounding box coordinates of the non-blank cells in the original table. The topological structure information includes the row and column starting positions and cross-row and column attributes of each cell.

[0017] Complete the missing values ​​in the row and column boundary matrix to generate the bounding box coordinates of the blank cells in the original table.

[0018] According to the table structure recognition method provided in this application, the row and column boundary matrix includes a first two-dimensional array and a second two-dimensional array.

[0019] The step of establishing a row and column boundary matrix based on the topological structure information of the original table and the bounding box coordinates of the non-blank cells in the original table includes:

[0020] Extract the left and right boundary coordinates of each column cell in the original table and construct a first two-dimensional array. In the first two-dimensional array, odd-numbered columns store the left boundary coordinates and even-numbered columns store the right boundary coordinates.

[0021] Extract the upper and lower boundary coordinates of each row cell in the original table and construct a second two-dimensional array. In the second two-dimensional array, the odd-numbered columns store the upper boundary coordinates and the even-numbered columns store the lower boundary coordinates.

[0022] The blank cells that are missing at least one bounding box coordinate are initialized with a specific null value identifier.

[0023] According to the table structure recognition method provided in this application, the step of filling in the missing values ​​in the row and column boundary matrix includes:

[0024] In the first two-dimensional array, for null identifiers representing the left boundary coordinates of columns, the minimum known value representing the left boundary coordinates in the column containing the null identifier is used as the padding value; and / or,

[0025] For null identifiers of the right boundary coordinates of a column, the maximum value of the known values ​​representing the right boundary coordinates in the column containing the null identifier is used as the padding value for padding.

[0026] According to the table structure recognition method provided in this application, the step of filling in the missing values ​​in the row and column boundary matrix includes:

[0027] If all values ​​representing the left boundary coordinates in the column containing null identifiers are null identifiers, then the maximum known value representing the right boundary coordinates in the previous column, plus a preset column width offset, is used as the padding value to pad all null identifiers representing the left boundary coordinates in that column; and / or,

[0028] If all values ​​representing the right boundary coordinates in the column containing null identifiers are null identifiers, then the minimum value of the known values ​​representing the left boundary coordinates in the next column minus a preset column width offset is used as the padding value to pad all null identifiers representing the right boundary coordinates in the column.

[0029] According to the table structure recognition method provided in this application, the step of filling in the missing values ​​in the row and column boundary matrix includes:

[0030] If the left / right boundary coordinates of multiple consecutive columns are null identifiers, then the average column width is determined based on the column width of the non-blank cells.

[0031] Using the right boundary coordinate of the column preceding the starting position of the sequence of consecutive null value identifiers as a reference, the left boundary coordinates / right boundary coordinates in the sequence of consecutive null value identifiers are sequentially supplemented by accumulating the average column width.

[0032] According to the table structure recognition method provided in this application, the step of filling in the missing values ​​in the row and column boundary matrix includes:

[0033] In the second two-dimensional array, for null identifiers of column upper boundary coordinates, the minimum known value representing the upper boundary coordinate in the column containing the null identifier is used as the padding value; and / or,

[0034] For null identifiers of column lower boundary coordinates, the maximum value of the known values ​​representing the lower boundary coordinates in the column containing the null identifier is used as the padding value for padding.

[0035] According to the table structure recognition method provided in this application, when a null value identifier appears in the first column / first row of the row and column boundary matrix, it is padded with zeros.

[0036] According to the table structure recognition method provided in this application, the step of fine-tuning the pre-trained table structure recognition model based on the training dataset formed by the original dataset and the augmented dataset includes:

[0037] The augmented dataset is mixed with the original dataset at a preset ratio;

[0038] The pre-trained basic model for table structure recognition is fine-tuned based on the mixed dataset.

[0039] According to the table structure recognition method provided in this application, after fine-tuning the pre-trained table structure recognition model based on the training dataset formed by the original dataset and the augmented dataset, the method further includes:

[0040] The table structure recognition performance of the trained basic model is evaluated based on a tree edit distance-based similarity method.

[0041] In a second aspect, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program that, when executed by the processor, implements the table structure recognition method as described in the first aspect.

[0042] Thirdly, a computer-readable storage medium is provided, wherein a table structure recognition program is stored on the computer-readable storage medium, and the table structure recognition program, when executed, implements any of the table structure recognition methods described in the first aspect above.

[0043] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the table structure recognition method as described in any of the first aspects above.

[0044] The table structure recognition method, device, and computer-readable storage medium described in this specification have the following advantages compared to the current methods that are difficult to effectively recognize wireless table structures:

[0045] By training a table structure recognition model capable of inferring the coordinates of blank cells, the table to be recognized is obtained. The table to be recognized is then input into the table structure recognition model, and the model automatically completes the inference. This achieves high-precision, complete, and consistent recognition of the wireless table structure of the table to be recognized, effectively solving the core technical problems in the existing technology, such as table structure breakage and row / column misjudgment caused by missing coordinates of blank cells.

[0046] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this specification. Attached Figure Description

[0047] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this specification and, together with the description, serve to explain the principles of this specification.

[0048] Figure 1 is a flowchart illustrating a table structure recognition method according to an exemplary embodiment of this specification.

[0049] Figure 2 is a flowchart illustrating the fine-tuning training of the table structure recognition model in a first embodiment of the table structure recognition method according to an exemplary embodiment of this specification.

[0050] Figure 3 is a schematic block diagram of a table structure recognition device according to an exemplary embodiment of this specification. Detailed Implementation

[0051] The technical solutions in the embodiments (or "implementations") of this application will be clearly and completely described herein with reference to the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements.

[0052] If the embodiments of this application contain terms relating to directional indications or positional relationships (such as up, down, left, right, front, back, inside, outside, top, bottom, center, vertical, horizontal, longitudinal, transverse, length, width, counterclockwise, clockwise, axial, radial, circumferential, etc.), such terms are only used to explain the relative positional relationships and movement of the components in a specific posture (as shown in the attached figures); if the specific posture changes, the directional indications or positional relationships will also change accordingly. Furthermore, the terms "first" and "second" used in the embodiments of this application are only for descriptive convenience and should not be construed as indicating or implying relative importance.

[0053] This application provides a method, apparatus, and computer-readable storage medium for recognizing table structures. The following detailed description, in conjunction with the accompanying drawings, illustrates this application. The features described in the embodiments and implementations can be combined with each other.

[0054] In wired tables, lines provide clear physical boundaries for all cells (including blank cells); however, in wireless tables, blank cells lack both text content and visual boundaries, causing the model to lose its dual anchor of "semantics" and "location," thus triggering a series of typical structural reasoning errors:

[0055] (1) Range misjudgment: Blank cells are incorrectly merged with adjacent cells, causing a deviation in the estimated total number of rows and columns of the table;

[0056] (2) Row and column break: consecutive blank cells are misjudged by the model as column break or table end mark, causing misalignment and breakage of the entire row and column relationship;

[0057] (3) Semantic loss: Key information such as missing values ​​in the experimental data were skipped directly, resulting in systematic bias in the downstream fine-tuning corpus.

[0058] Therefore, there is an urgent need for a method that can effectively identify wireless table structures to solve this problem.

[0059] To address the aforementioned technical issues, this specification provides a method for identifying table structures.

[0060] The aim is to improve the performance of table structure recognition models by completing the coordinates of blank cells through structural reasoning. By accurately reasoning the geometric position of blank cells through the table structure recognition model, the overall accuracy and robustness of table recognition are improved.

[0061] This application provides a first embodiment of a table structure recognition method. Referring to FIG1, FIG1 is a schematic flowchart of a first embodiment of a table structure recognition method provided in this specification.

[0062] In some embodiments, the method specifically includes the following steps 101 to 102:

[0063] Step 101: Obtain the table to be recognized;

[0064] Step 102: Input the table to be identified into a preset table structure recognition model and output the table structure information of the table to be identified;

[0065] The table structure recognition model is trained using a training dataset containing blank cells. The trained table structure recognition model can identify blank cells in the table to be recognized and output complete table structure information containing blank cells.

[0066] The table to be identified includes at least blank cells and non-blank cells.

[0067] The table to be identified is input into the preset table structure recognition model. The table structure recognition model can infer the position information of blank cells and combine it with the position information of non-blank cells to identify the table structure information of the table to be identified.

[0068] In some embodiments, referring to FIG2, FIG2 is a flowchart illustrating the fine-tuning training of a table structure recognition model in a first embodiment of a table structure recognition method according to an exemplary embodiment of this specification.

[0069] The table structure recognition model is trained using a training dataset containing blank cells. Step 102 includes the following steps 1021 to 1023:

[0070] Step 1021: Obtain the original dataset corresponding to the original table containing blank cells and non-blank cells.

[0071] This paper provides a method for automatically generating high-quality table structure training data. By filling in the geometric coordinates of blank cells, the table structure recognition model trained on this data has significantly improved its recognition accuracy and generalization ability for the overall table structure (especially the structure containing blank cells).

[0072] First, obtain the original dataset of the original table containing blank cells and non-blank cells. The blank cells include single blank cells, merged blank cells, or entire rows and columns of blank cells.

[0073] As an example, the original dataset of the original table includes a screenshot of the table and labeled data. The labeled data includes the table structure (HTML tags), the content of each cell, and the bounding box of each cell. The labeled data containing blank cells is characterized by the fact that the content of the blank cells is empty and the bounding box information of the blank cells is empty.

[0074] Step 1022: Based on the topological structure information of the original table and the bounding box coordinates of the non-blank cells in the original table, complete the bounding box coordinates of the blank cells in the original table, and generate an augmented dataset including the bounding box coordinates of the completed blank cells.

[0075] For training data in a table containing blank cells, the bounding box of the blank cells is calculated using a cell completion algorithm.

[0076] In some embodiments, step 1022, which involves filling in the bounding box coordinates of blank cells in the original table based on the topological information of the original table and the bounding box coordinates of non-blank cells in the original table, includes steps a1 to a2:

[0077] Step a1: Establish a row and column boundary matrix based on the topological structure information of the original table and the bounding box coordinates of the non-blank cells in the original table. The topological structure information includes the row and column starting positions and cross-row and column attributes of each cell.

[0078] As an example, the topological information of the original table defines the logical grid of the table, including which row and column a cell is located in, how many rows and columns a cell occupies, etc.

[0079] For example, the topology information is represented as {'start_row': int,'start_col': int,'col_span': int,'row_span': int}.

[0080] As an example, the bounding box coordinates of a non-blank cell in the original table refer to the cell's position within the original table. Typically, the bounding box of each cell is represented by [x1, y1, x2, y2], where [x1, y1] represents the coordinates of the top-left corner of the cell, and [x2, y2] represents the coordinates of the bottom-right corner of the cell.

[0081] Extract the bounding box coordinates of all cells in the original table to create a row and column boundary matrix.

[0082] The row and column boundary matrix includes a first two-dimensional array and a second two-dimensional array. By creating a two-dimensional array, the problem of reasoning the coordinates of blank cells based on the coordinates of adjacent cells is reduced from two-dimensional relational reasoning to one-dimensional sequence completion, thus achieving systematic and batch processing.

[0083] The one-dimensional problem includes row boundary problems and column boundary problems. By constructing a first two-dimensional array and a second two-dimensional array, the structural information of the table is extracted into a regular data structure. The completion algorithm only needs to process the NaN values ​​in these arrays, without needing to know which cell each NaN belongs to in the original table. A uniform, serialized completion rule can be applied to the entire array, resulting in a simpler and more computationally efficient solution.

[0084] In some embodiments, establishing a row and column boundary matrix based on the topological structure information of the original table and the bounding box coordinates of the non-blank cells in the original table includes:

[0085] Extract the left and right boundary coordinates of each column cell in the original table and construct a first two-dimensional array. In the first two-dimensional array, odd-numbered columns store the left boundary coordinates and even-numbered columns store the right boundary coordinates.

[0086] Extract the upper and lower boundary coordinates of each row cell in the original table and construct a second two-dimensional array. In the second two-dimensional array, the odd-numbered columns store the upper boundary coordinates and the even-numbered columns store the lower boundary coordinates.

[0087] The blank cells that are missing at least one bounding box coordinate are initialized with a specific null value identifier.

[0088] As an example, a specific null value identifier can be NaN.

[0089] Extract the x1 and x2 information of all cells in the original table, initialize the x1 and x2 values ​​of each column of the table, and initialize blank cells with NaN if they do not have x1 and x2 values, forming an n*2m first two-dimensional array, where n is the number of rows in the table and m is the number of columns in the table.

[0090] Extract the y1 and y2 information of all cells in the original table, initialize the y1 and y2 values ​​of each row of the table, and initialize blank cells with NaN if they do not have y1 and y2 values, forming an m*2n second two-dimensional array, where n is the number of rows in the table and m is the number of columns in the table.

[0091] Step a2: Complete the missing values ​​in the row and column boundary matrix to generate the bounding box coordinates of the blank cells in the original table.

[0092] The following is a method for filling blank cells based on the first two-dimensional array.

[0093] First, when a null value identifier appears in the first column / row of the row and column boundary matrix, it is padded with zeros. The first column typically starts from the document edge, using 0 as the default starting point to prevent negative coordinates.

[0094] Next, for the other odd and even columns in the first two-dimensional array, the NaN values ​​are filled in by iterative processing using the minimum / maximum values ​​of the known coordinates, ensuring the continuity and consistency of the table structure.

[0095] As an example, the process of filling in missing values ​​in the row and column boundary matrix includes:

[0096] In the first two-dimensional array, for null identifiers of column left boundary coordinates, the minimum known value representing the left boundary coordinate in the column where the null identifier is located is used as the padding value for padding.

[0097] In practical use, the data in the table contains noise and errors, and the boundary coordinates of the same column are not completely consistent. For example, in real OCR output, due to image distortion, printing offset, or recognition errors, even cells in the same column may have slight fluctuations in their boundary coordinates.

[0098] Therefore, by taking the minimum known value representing the left boundary coordinate in the column as the padding value, we can ensure that the left boundary of the padding cell is aligned with the leftmost cell in the table, thus avoiding overlap between the padding box and any existing box.

[0099] Similarly, for null identifiers of the right boundary coordinates of a column, the maximum value of the known values ​​representing the right boundary coordinates in the column containing the null identifier is used as the padding value for padding.

[0100] In other words, by taking the maximum value of the known values ​​representing the right boundary coordinates in the column as the padding value, the right boundary of the padding cell B2 can be wide enough to completely cover the content area of ​​the column, avoiding the inability to include text that may be recognized in the future due to the box being too narrow.

[0101] In other words, in real data with coordinate fluctuations, the min(x1) strategy guarantees non-overlap, while the max(x2) strategy guarantees coverage.

[0102] In some embodiments, the process of filling in missing values ​​in the row and column boundary matrix includes:

[0103] If all values ​​representing the left boundary coordinates in the column containing null identifiers are null identifiers, then the maximum value of the known values ​​representing the right boundary coordinates in the previous column plus a preset column width offset is used as the padding value to pad all null identifiers representing the left boundary coordinates in the column.

[0104] If the entire column is NaN, it means that the column is completely blank. The position of the current column is then calculated based on the boundary coordinates of the adjacent columns.

[0105] Normally, the left and right edges of a table cell do not fit together visually. If the left and right edges have the same coordinates, or if the width is 0 pixels during rendering, it does not conform to the visual appearance of a table. At the same time, giving the model a cell with an unreasonable width (such as the left and right edges having the same coordinates) as a training sample will also interfere with the model's learning of normal column widths.

[0106] Therefore, a preset column width offset is set to ensure that the newly added column has a non-zero, reasonable minimum width, thus clearly separating it from the previous column in geometric space.

[0107] As an example, the preset column width offset can be, but is not limited to, a fixed small integer (such as 1 pixel), a percentage of the average or median of all known column widths (such as 10%), or a tiny percentage of the image width.

[0108] For example, if the entire odd column k is NaN, then the maximum value of column k-1 plus 1 is used to fill in the NaN values ​​in column k.

[0109] Similarly, if all the values ​​representing the right boundary coordinates in the column containing null identifiers are null identifiers, then the minimum value of the known values ​​representing the left boundary coordinates in the next column minus a preset column width offset is used as the padding value to pad all the null identifiers representing the right boundary coordinates in the column.

[0110] For example, if the entire even-numbered column q is NaN, then the minimum value of column q+1 minus 1 is used to fill in the NaN values ​​in column q.

[0111] In some embodiments, the process of filling in missing values ​​in the row and column boundary matrix includes:

[0112] If the left / right boundary coordinates of multiple consecutive columns are null identifiers, then the average column width is determined based on the column width of the non-blank cells.

[0113] Using the right boundary coordinate of the column preceding the starting position of the sequence of consecutive null value identifiers as a reference, the left boundary coordinates / right boundary coordinates in the sequence of consecutive null value identifiers are sequentially supplemented by accumulating the average column width.

[0114] When multiple consecutive columns are blank, since there is no reliable data to refer to from adjacent cells, it is assumed that the widths of all columns are roughly uniform or continuous. Based on the assumption of continuity in the overall table layout, the widths of unknown columns are inferred from the known column widths.

[0115] Specifically, assume that columns p through q are all NaN. The algorithm will traverse the entire table and calculate the average of the widths w = (x2 - x1) of all known and valid columns.

[0116] First, complete the p-th column (odd-numbered columns), with its left boundary x1. p x2 is considered to be the right boundary of the previous column (the (p-1)th column, whose boundary coordinates have been determined). p-1 Its right boundary x2 p Then through x1 p The result is calculated using +w.

[0117] Next, the left boundary x1 of column p+1 p+1 It equals x2 p Right boundary x2 p+1 =x1p+1 +w.

[0118] Continue in this manner until all consecutive blank columns are filled.

[0119] The above process completes the values ​​representing cell positions in the first two-dimensional array. The same algorithm is used to complete the second two-dimensional array until neither the first nor the second two-dimensional array contains NaN values.

[0120] In some embodiments, the process of filling in missing values ​​in the row and column boundary matrix includes:

[0121] In the second two-dimensional array, for null identifiers of column upper boundary coordinates, the minimum known value representing the upper boundary coordinate in the column containing the null identifier is used as the padding value; and / or,

[0122] For null identifiers of column lower boundary coordinates, the maximum value of the known values ​​representing the lower boundary coordinates in the column containing the null identifier is used as the padding value for padding.

[0123] The process of filling in the missing values ​​in the row and column boundary matrix includes:

[0124] If all values ​​representing the upper boundary coordinates in the column containing null identifiers are null identifiers, then the maximum value of the known values ​​representing the lower boundary coordinates in the previous column, plus a preset column width offset, is used as the padding value to pad all null identifiers representing the upper boundary coordinates in that column; and / or,

[0125] If all values ​​representing the lower boundary coordinates in the column containing null identifiers are null identifiers, then the minimum value of the known values ​​representing the upper boundary coordinates in the next column minus a preset column width offset is used as the padding value to pad all null identifiers representing the lower boundary coordinates in the column.

[0126] The process of filling in the missing values ​​in the row and column boundary matrix includes:

[0127] If the upper / lower boundary coordinates of multiple consecutive columns are null identifiers, then the average column width is determined based on the column width of the non-blank cells;

[0128] Using the lower boundary coordinates of the column preceding the starting position of the sequence of consecutive null value identifiers as a reference, the upper boundary coordinates / lower boundary coordinates in the sequence of consecutive null value identifiers are sequentially supplemented by accumulating the average column width.

[0129] The second two-dimensional array is completed using the same algorithm as the first two-dimensional array, which will not be elaborated here.

[0130] Through the above process, all cells have accurate coordinate information, and the model can learn the complete table structure.

[0131] Step 1023: Fine-tune the pre-trained table structure recognition model based on the training dataset formed by the original dataset and the augmented dataset.

[0132] After the coordinates of the blank cells were filled in using an algorithm, the data was cleaned to remove bad samples generated during the filling process, thus ensuring the quality of the enhanced dataset.

[0133] As an example, data cleaning may include, but is not limited to, coordinate validity checks, coordinate out-of-bounds checks, logical consistency checks, and conflict resolution.

[0134] The coordinate validity check refers to verifying the validity of all completed coordinates [x1, y1, x2, y2]. For example, it ensures that x2 > x1 and y2 > y1 to prevent invalid boxes with negative widths or heights.

[0135] Coordinate boundary checks ensure that the completed bounding box does not exceed the image boundaries. If it does, cropping or removal of the sample is necessary.

[0136] Logical consistency checks utilize topological information to inspect the completed table for severe row and column overlaps or misalignments. For example, the y-coordinate range of cells in the same row should be approximately the same, and the x-coordinate range of cells in the same column should be approximately the same.

[0137] Conflict resolution refers to the process where, if the completed box overlaps with an existing non-empty cell box, a decision is made based on the overlap area and confidence level. Typically, the original true label is preserved first, and the completed bounding box is adjusted.

[0138] After data cleaning, the cleaned data is converted into the input format required by a specific deep learning framework.

[0139] Subsequently, a flag is added to the dataset metadata to explicitly identify which data points are augmented data. This helps track the impact of different data sources on model performance in subsequent experimental analyses.

[0140] Through the above process, a high-quality, uniformly formatted augmented dataset is generated.

[0141] Finally, the pre-trained table structure recognition model was fine-tuned using the training dataset formed by the original dataset and the augmented dataset.

[0142] In some embodiments, fine-tuning the pre-trained table structure recognition model based on the training dataset formed by the original dataset and the augmented dataset includes:

[0143] The augmented dataset is mixed with the original dataset at a preset ratio;

[0144] The pre-trained basic model for table structure recognition is fine-tuned based on the mixed dataset.

[0145] The augmented dataset is matched with a normal dataset, and a basic table structure recognition model pre-trained on a large tabular dataset is selected as the starting point. Using the mixed dataset, the parameters of the pre-trained basic table structure recognition model are updated, resulting in a fine-tuned basic table structure recognition model that gains the powerful ability to handle blank cells.

[0146] In some embodiments, after fine-tuning the pre-trained table structure recognition model using the training dataset formed by the original dataset and the augmented dataset, the method further includes:

[0147] The table structure recognition performance of the trained basic model is evaluated based on a tree edit distance-based similarity method.

[0148] For the trained table structure recognition baseline model, the TEDS method is used for evaluation. The TEDS score measures the accuracy of the model's predictions. The higher the score, the closer the model's predicted HTML structure and content are to the ground truth annotations. This allows for an objective and quantitative evaluation of the fine-tuned model's performance, demonstrating the effectiveness of the completion scheme.

[0149] The specific process is as follows:

[0150] Step 1: Obtain open-source data for training the table structure recognition model, using Pubtabnet as an example. This open-source dataset contains labeled data for approximately 500,000 tables, the vast majority of which are non-linear tables. Approximately 200,000 data points in this dataset contain blank cells.

[0151] The format of the labeled data is mainly as follows:

[0152] filename: Image name;

[0153] split: train / val / test;

[0154] html: The complete HTML string, provided.

[0155] token: The text content of the cell;

[0156] bbox: [x1,y1,x2,y2] The bounding box coordinates of the cell.

[0157] When a cell is empty, the token field is empty and the bonding box field is missing.

[0158] Step 2: Use the table cell completion algorithm to complete the table data containing blank cells in the Pubtabnet dataset and calculate the missing bounding box values.

[0159] Step 3: Convert the format of the completed data to the format matched by the paddleOCR model training framework.

[0160] Step 4: Select two datasets for comparison and verification.

[0161] Dataset A consists of 200,000 randomly selected data points from Pubtabnet, including blank cells.

[0162] Given dataset A', fill in all blank cells in the tabular data within dataset A. Then, combine the original blank cells with the filled cells to generate dataset A'.

[0163] Step 5: Use the paddleOCR model training framework to fine-tune the pre-trained table structure recognition model using datasets A and A' respectively, to obtain two table structure recognition models, namely the first table structure recognition model and the second table structure recognition model (i.e., the model used by this method to recognize the table structure of the table to be recognized).

[0164] Step 6: Select 500 test data entries from PubTabNet and 500 test data entries from FinTabNet to evaluate the model's inference performance. FinTabNet is a financial table dataset, belonging to a different domain than PubTabNet (general academic tables). It is used to test the generalization ability of the table recognition model. Using FinTabNet as the test set aims to verify whether the model trained on PubTabNet data can effectively recognize previously unseen financial tables from a different domain.

[0165] TEDS-Structure and TEDS-Overall scores were calculated for 500 test data points each from Pubtabnet and Fintabnet. The TEDS-Structure score characterizes the model's ability to recognize row and column structures and cell merging relationships. The TEDS-Overall score evaluates the model's overall output capability.

[0166] The Teds method was used to evaluate the performance of the two models, and the average scores are shown in the table below:

[0167]

[0168] As shown in the table above, the second table structure recognition model performs better than the first, proving that completion training is effective on similar data. Furthermore, it demonstrates that the generalization ability of the proposed model is also improved through completion training.

[0169] Based on this, the second table structure recognition model obtained by this method can accurately identify the table structure information of the input table to be recognized.

[0170] This application provides a table structure recognition method, device, and computer-readable storage medium, which has the following advantages compared with the current methods that are difficult to effectively recognize wireless table structures:

[0171] By training a table structure recognition model capable of inferring the coordinates of blank cells, the table to be recognized is obtained. The table to be recognized is then input into the table structure recognition model, and the model automatically completes the inference. This achieves high-precision, complete, and consistent recognition of the wireless table structure of the table to be recognized, effectively solving the core technical problems in the existing technology, such as table structure breakage and row / column misjudgment caused by missing coordinates of blank cells.

[0172] Based on the same concept as the methods described above, this application also proposes a table structure recognition system. The system includes:

[0173] The table acquisition module is used to acquire the table to be recognized;

[0174] The table structure recognition module is used to input the table to be recognized into a preset table structure recognition model and output the table structure information of the table to be recognized.

[0175] The table structure recognition model is trained using a training dataset containing blank cells. The trained table structure recognition model can identify blank cells in the table to be recognized and output complete table structure information containing blank cells.

[0176] The specific implementation process of the functions and roles of each module in the above device can be found in the implementation process of the corresponding steps in the above method, which can achieve the same technical effect, and will not be repeated here.

[0177] Figure 3 illustrates a schematic diagram of the physical structure of a table structure recognition device. As shown in Figure 3, the table structure recognition device may include: a processor 510, a communication interface 520, a memory 530, and a communication bus 540. The processor 510, communication interface 520, and memory 530 communicate with each other via the communication bus 540. The processor 510 can call logical instructions in the memory 530 to execute the table structure recognition method.

[0178] Furthermore, the logical instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0179] On the other hand, this application also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the table structure recognition method provided by the above methods.

[0180] In another aspect, this application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the table structure recognition methods provided by the above methods.

[0181] It should be noted that the technical solutions or features described in the above embodiments can be combined or supplemented with each other without conflict. The scope of protection of this application is not limited to the precise structures described in the above embodiments and shown in the accompanying drawings; all modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A method for recognizing table structures, characterized in that, The method includes: acquiring a table to be identified; inputting the table to be identified into a preset table structure recognition model, and outputting the table structure information of the table to be identified; wherein, the table structure recognition model is trained using a training dataset containing blank cells, and the trained table structure recognition model can identify blank cells in the table to be identified and output complete table structure information containing blank cells; the table structure recognition model is trained using a training dataset containing blank cells, including: acquiring an original dataset corresponding to an original table containing blank cells and non-blank cells; based on the topological structure information of the original table and the bounding box coordinates of the non-blank cells in the original table, completing the bounding box coordinates of the blank cells in the original table, and generating an augmented dataset including the bounding box coordinates of the completed blank cells; fine-tuning the pre-trained table structure recognition model based on the training dataset formed by the original dataset and the augmented dataset.

2. The table structure recognition method as described in claim 1, characterized in that, The step of filling in the bounding box coordinates of blank cells in the original table based on the topological structure information of the original table and the bounding box coordinates of non-blank cells in the original table includes: establishing a row and column boundary matrix based on the topological structure information of the original table and the bounding box coordinates of non-blank cells in the original table, wherein the topological structure information includes the row and column starting position and cross-row and column attributes of each cell; filling in the missing values ​​in the row and column boundary matrix to generate the bounding box coordinates of blank cells in the original table.

3. The table structure recognition method as described in claim 2, characterized in that, The row and column boundary matrix includes a first two-dimensional array and a second two-dimensional array. The step of establishing the row and column boundary matrix based on the topological structure information of the original table and the bounding box coordinates of the non-blank cells in the original table includes: extracting the left and right boundary coordinates of each column cell in the original table to construct a first two-dimensional array, where odd-numbered columns store left boundary coordinates and even-numbered columns store right boundary coordinates; extracting the upper and lower boundary coordinates of each row cell in the original table to construct a second two-dimensional array, where odd-numbered columns store upper boundary coordinates and even-numbered columns store lower boundary coordinates; wherein, for blank cells lacking at least one bounding box coordinate, the bounding box coordinates are initialized using a specific null value identifier.

4. The table structure recognition method as described in claim 3, characterized in that, The step of filling in the missing values ​​in the row and column boundary matrix includes: in the first two-dimensional array, for a null identifier of the left boundary coordinate of a column, filling in the missing value by using the minimum known value representing the left boundary coordinate in the column where the null identifier is located as the fill value; and / or, for a null identifier of the right boundary coordinate of a column, filling in the missing value by using the maximum known value representing the right boundary coordinate in the column where the null identifier is located as the fill value.

5. The table structure recognition method as described in claim 3, characterized in that, The method of filling in the missing values ​​in the row and column boundary matrix includes: if all the values ​​representing the left boundary coordinates in the column containing the null identifier are null identifiers, then the maximum value of the known values ​​representing the right boundary coordinates in the previous column plus a preset column width offset is used as the fill value to fill in all the null identifiers representing the left boundary coordinates in the column; and / or, if all the values ​​representing the right boundary coordinates in the column containing the null identifier are null identifiers, then the minimum value of the known values ​​representing the left boundary coordinates in the next column minus a preset column width offset is used as the fill value to fill in all the null identifiers representing the right boundary coordinates in the column.

6. The table structure recognition method as described in claim 3, characterized in that, The process of filling in the missing values ​​in the row and column boundary matrix includes: if the left boundary coordinates / right boundary coordinates of multiple consecutive columns are null identifiers, then the average column width is determined based on the column width of the non-blank cells; and the left boundary coordinates / right boundary coordinates in the sequence of consecutive null identifiers are filled in sequentially by adding the average column width, using the right boundary coordinate of the column preceding the starting position of the sequence of consecutive null identifiers as a reference.

7. The table structure recognition method as described in claim 3, characterized in that, The process of filling in the missing values ​​in the row and column boundary matrix includes: in the second two-dimensional array, for a null identifier of the upper boundary coordinate of a column, filling in the missing value by using the minimum known value representing the upper boundary coordinate in the column where the null identifier is located as the filling value; and / or, for a null identifier of the lower boundary coordinate of a column, filling in the missing value by using the maximum known value representing the lower boundary coordinate in the column where the null identifier is located as the filling value.

8. The table structure recognition method as described in any one of claims 4 to 7, characterized in that, When a null identifier appears in the first column / first row of the row and column boundary matrix, it is padded with zeros.

9. The table structure recognition method as described in claim 1, characterized in that, The step of fine-tuning the pre-trained table structure recognition model using a training dataset formed by the original dataset and the augmented dataset includes: mixing the augmented dataset and the original dataset at a preset ratio; and fine-tuning the pre-trained table structure recognition model using the mixed dataset.

10. The table structure recognition method as described in claim 1, characterized in that, After fine-tuning the pre-trained table structure recognition model using the training dataset formed by the original dataset and the augmented dataset, the method further includes: evaluating the table structure recognition performance of the trained table structure recognition basic model based on a similarity method based on tree edit distance.

11. An electronic device comprising a memory and a processor, wherein the memory stores a computer program that, when executed by the processor, implements the table structure recognition method as described in any one of claims 1 to 10.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed, implements the table structure recognition method as described in any one of claims 1 to 10.

Images