A method for evaluating the accuracy of table structure recognition

Abstract

The present invention discloses a method for evaluating the accuracy of table structure recognition. The present invention uses a markup language to formally represent the table structure of the table, and the table structure obtained before the table is recognized is recorded as the true table structure; the table structure of the table is identified using a table structure prediction model and recorded as the predicted table structure; according to the true table structure and the predicted table structure of the table, a horizontal line formalized binary matrix and a vertical line formalized binary matrix are output respectively; the binary matrix of the true table structure is used as a template matrix, and the binary matrix of the predicted table is used as a prediction matrix; each template matrix is moved in its corresponding prediction matrix to find the best matching position area of the template matrix in the prediction matrix; based on the best matching position area, each element in the template matrix and the prediction matrix is matched and compared to obtain a confusion matrix to calculate the table structure recognition accuracy evaluation index, which is used to evaluate the prediction model. The present invention greatly improves the accuracy of prediction model evaluation.

Description

Technical Field

[0001] The present invention relates to a method for evaluating the accuracy of table structure recognition, which aims to evaluate the performance of table structure recognition tasks and belongs to the field of human-computer interaction. Background Art

[0002] Tables have long been widely used in various application scenarios as an effective way to organize and present structured data. Due to their inherent structured nature, they have become a common form of data display and statistics in various electronic documents. They can also be used in hand-drawn sketches as the most natural and intuitive medium for communicating ideas, organizing inspiration, and concretizing concepts. Therefore, table recognition has gradually become a research hotspot for researchers both domestically and internationally.

[0003] In the definition of table recognition tasks, Hu et al. (reference: Hu J, Kashi RS, Lopresti D, et al. Evaluating the performance of table processing algorithms [J]. International Journal on Document Analysis and Recognition, 2002, 4 (3): 140-153.) divided the table recognition task into two more fine-grained subtasks: table detection and table recognition. Among them, the table detection task aims to detect the location area of the table from a page, which is also called layout analysis and is often classified as a target detection task in computer vision. The table structure recognition task is to further identify the logical structure and content information of the table based on the table location area detected by the table detection task, extract the internal logical structure information such as the rows, columns, and cells of the table, and perform character recognition on the content in the cell through optical character recognition technology. Finally, the recognized and reconstructed table can be displayed and stored through different visualization forms and storage technologies. The present invention focuses more on the recognition effect of the table structure. Therefore, the present invention further refines the definition of the table recognition task, and defines the purpose of the table structure recognition task as identifying the row and column layout structure of the table. Specifically, the output of table structure recognition is the logical structure of the table, including the number of rows and columns, the logical relationships between rows and columns, and cell attributes. Cell attributes include the relative coordinates of the cell's location, whether it is a complex cell, and the number of rows and columns each cell spans.

[0004] In recent years, with the rapid development of deep learning, many deep learning-based methods have emerged. Researchers have applied deep learning methods such as object detection, image semantic segmentation, and graph neural networks to the task of table structure recognition. Lin et al. (Reference: Lin W, Sun Z, Ma C, et al. TSRFormer: Table Structure Recognition with Transformers [C] / / Proceedings of the 30th ACM International Conference on Multimedia. 2022: 6473-6482.) defined the table structure recognition task as a predictive regression problem for table separator line detection. In this paper, the F1-score metric is used to evaluate the performance of the model algorithm in table structure recognition. F1-score is an indicator that comprehensively evaluates the classification performance of the model. The calculation formula is: F1-score = 2*(precision*recall) / (precision+recall), where precision is the accuracy rate, which represents the ratio of the number of correctly predicted table dividers to the number of all predicted table dividers, and recall is the recall rate, which represents the ratio of the number of correctly predicted table dividers to the number of all true table dividers. This evaluation method is suitable for scenarios where the internal recognition of the table structure is incorrect, that is, the overall structure of the table is the same, but the attribute recognition of the internal cells is incorrect. However, in scenarios where the overall structure of the table is incorrectly recognized, this evaluation method is prone to fall into the local optimum when comparing the two table structures, resulting in a decrease in the recognition index obtained by the evaluation. Summary of the Invention

[0005] In response to the technical problems existing in the prior art, the purpose of the present invention is to provide a method for evaluating the accuracy of table structure recognition. First, a markup language is selected to formally represent the table structure, and secondly, the method proposed by the present invention is applied to evaluate the performance of the table structure recognition task. In the table structure recognition task, the present invention avoids the problem that the traditional evaluation method, which is prone to reduce the evaluation index and ultimately lead to inaccurate evaluation in the scenario where the overall structure of the table is incorrectly recognized (the total number of rows and columns is different), by finding the step of finding the global optimal match between the two table structures.

[0006] In the present invention, the steps for calculating the table structure evaluation index before and after the two recognitions are as follows, see reference Figure 1 For the sake of unification and understanding in the following text, the present invention refers to the two table structures before and after recognition as the “real table structure” and the “predicted table structure” respectively.

[0007] Step 1: Based on the structured markup language of the actual table structure and the predicted table structure, a binary matrix representing the formalization of horizontal and vertical lines is generated, indicating whether the horizontal and vertical lines that constitute each cell in the table exist. If a horizontal or vertical line exists, the value of the element at that position in the matrix is 1; otherwise, it is 0. For the steps for converting the formalized table structure into the binary matrix representing the formalization of horizontal and vertical lines, please refer to the specific implementation plan.

[0008] Step 2: Use the horizontal and vertical formalized binary matrices of the actual table structure as template matrices, namely the horizontal template matrix and the vertical template matrix, respectively. Use the horizontal and vertical formalized binary matrices of the predicted table as prediction matrices, namely the horizontal prediction matrix and the vertical prediction matrix, respectively. Slide each template matrix within its corresponding prediction matrix, traversing the entire prediction matrix from left to right and from top to bottom, with a step size of 1, until the best matching position region of the template matrix in the prediction matrix is found, calculated using the "accuracy of each matching position region" as the calculation criterion. The "accuracy of each matching position region" is defined as the percentage of the number of elements in the template matrix (generated by each round of sliding) that share the same corresponding position in each matching position region as the prediction matrix, divided by the total number of elements in the template matrix. Specifically, "accuracy of each matching position region" = (number of elements in the template matrix (number of elements in the template matrix (number of elements in the prediction matrix (number of elements in the template matrix (number of elements in the prediction matrix (number of elements in the template matrix (number of elements in the prediction matrix (number of elements in the template matrix (number of elements in the prediction matrix ( For details about the sliding process, please refer to the specific implementation plan. The matching position area refers to a submatrix area in the prediction matrix, which is the area where the template matrix slides within the prediction matrix at intervals of 1. When the template matrix is a horizontal line template matrix, the corresponding prediction matrix is the horizontal line prediction matrix; when the template matrix is a vertical line template matrix, the corresponding prediction matrix is the vertical line prediction matrix.

[0009] Step 3: For the two horizontal-line formalized binary matrices of the actual table structure and the predicted table structure, at the best matching location of the template matrix (the horizontal-line formalized binary matrix of the actual table structure) in the predicted matrix (the horizontal-line formalized binary matrix of the predicted table structure), each element in the template matrix and the predicted matrix is matched and compared to obtain a 2×2 confusion matrix, ConfusionMatrix4Row. For the two vertical-line formalized binary matrices of the actual table structure and the predicted table structure, at the best matching location of the template matrix (the vertical-line formalized binary matrix of the actual table structure) in the predicted matrix (the vertical-line formalized binary matrix of the predicted table structure), each element in the template matrix and the predicted matrix is matched and compared to obtain a 2×2 confusion matrix, ConfusionMatrix4Col. Finally, the confusion matrix, ConfusionMatrix4Row, obtained by matching and comparing the two horizontal-line formalized binary matrices, is combined with the confusion matrix, ConfusionMatrix4Col, obtained by matching and comparing the two vertical-line formalized binary matrices to obtain the final 2×2 confusion matrix, ConfusionMatrix.

[0010] Step 4: Based on the final confusion matrix, objective indicators such as accuracy, precision, recall, and F1-score are calculated to evaluate the performance of the prediction model algorithm in the table structure recognition task.

[0011] The present invention also provides a server, characterized in that it includes a memory and a processor, the memory stores a computer program, the computer program is configured to be executed by the processor, and the computer program includes instructions for executing each step in the above method.

[0012] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps of the above method when executed by a processor.

[0013] The advantages of the present invention are as follows:

[0014] (1) A table recognition structure accuracy evaluation method is proposed to evaluate the performance of table structure recognition tasks.

[0015] (2) For the table structure recognition task, the present invention avoids the problem that the traditional evaluation method easily leads to a decrease in evaluation index and ultimately inaccurate evaluation when the overall structure of the table is incorrectly recognized (the total number of rows and columns is different) by finding the global optimal match of the two table structures.

[0016] (3) A method is proposed to compare the elements between two matrices through the formal expression of binary matrices to quickly calculate the evaluation indicators of the table structure. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a flow chart of the calculation steps of the table structure comparison index before and after two recognitions.

[0018] Figure 2 This is an example diagram of the construction of a binary matrix formalized by horizontal and vertical lines in a tabular structure.

[0019] Figure 3 This is an example diagram of the process of a template matrix sliding in the prediction matrix. DETAILED DESCRIPTION

[0020] The present invention will be described in further detail below with reference to the accompanying drawings. The examples given are only used to explain the present invention and are not used to limit the scope of the present invention.

[0021] In Step 1, the present invention uses HTML hypertext markup language to formally represent the table structure. The steps for constructing two formal binary matrices representing horizontal and vertical lines, respectively, for a table with m rows and n columns are as follows:

[0022] (1) First, define two matrices RM and CM of dimensions (m+1)×n and m×(n+1), respectively. RM represents the formalized binary matrix of horizontal lines, and CM represents the formalized binary matrix of vertical lines. Here, m and n represent the number of rows and columns of the table, respectively. RM (the formalized binary matrix of horizontal lines) expresses whether a horizontal line exists in each cell of the table. For a table with m rows and n columns, there are m+1 rows of horizontal lines; the same applies to CM.

[0023] (2) Secondly, assign values to all elements in the horizontal line formalization binary matrix RM and the vertical line formalization binary matrix CM respectively to indicate whether the horizontal and vertical lines that constitute each cell in the table exist. For the horizontal line formalization binary matrix RM, for each element position of the matrix, if there is a horizontal line at the corresponding cell position, the element value of the position is 1, otherwise it is 0. For the vertical line formalization binary matrix CM, for each element position of the matrix, if there is a vertical line at the corresponding cell position, the element value of the position is 1, otherwise it is 0. As shown in reference Figure 2As shown in the figure, for a table with complex cells (cells that span more than one row or column), first, the complex cells are divided into meta-cells (that is, cells that only span one row and one column and cannot be divided any further are called meta-cells), and the horizontal or vertical lines used for division are distinguished by dotted lines. Next, all the dotted line positions are mapped one by one to the corresponding positions in RM and CM, and the element values at the dotted position are assigned to 0, and the element values at other positions are assigned to 1.

[0024] In Step 2, the core operation is to slide each template matrix groundTruthMatrix in its corresponding prediction matrix predictMatrix at intervals of 1, traversing the entire prediction matrix from left to right and from top to bottom. Specifically, the size of the template matrix and the prediction matrix may not be the same. This is because if a prediction model performs poorly, the predicted table structure may have x more or x fewer rows or columns than the actual table structure. In this case, the size of the prediction matrix will be different from the size of the template matrix, so the template matrix needs to be moved in the prediction matrix. The specific sliding process can be simply described as the following steps:

[0025] (1) Align the lower right corner element of the template matrix with the upper left corner element of the prediction matrix.

[0026] (2) If the size of the submatrix formed by the matching position area of the template matrix in the prediction matrix is the same as the size of the template matrix, the accuracy of the current matching position area is calculated according to the "accuracy of each matching position area"; if the size of the submatrix formed by the matching position area is smaller than the size of the template matrix, the vacant positions in the prediction matrix are filled with elements 0 to obtain the matching position area of the template matrix in the prediction matrix, and the accuracy of the current matching position area is calculated according to the "accuracy of each matching position area". (3) Move the template matrix to the right by the distance of one element position, and then repeat step 2 until the template matrix traverses the width of the prediction matrix. Interval movement with a step size of 1 refers to moving one element in the formalized binary matrix. When the table structure is formalized into a binary matrix, a cell is composed of two horizontal lines and two vertical lines, corresponding to two adjacent elements in the horizontal line binary matrix and the vertical line binary matrix respectively.

[0027] (4) Move the template matrix downward by the distance of one element position, and then repeat steps 2 and 3 until the template matrix traverses the height of the prediction matrix.

[0028] Based on this rule, for a template matrix groundTruthMatrix with dimensions m1×n1 and a prediction matrix predictMatrix with dimensions m2×n2, the total number of walks is (m1+m2-1)×(n1+n2-1). For example, the walk-sliding process of a template matrix groundTruthMatrix with dimensions 2×3 in a prediction matrix with dimensions 4×5 is shown in the reference. Figure 3 As shown. By sliding the template matrix in sequence into the submatrix composed of the matching position areas of the prediction matrix, the present invention finds the best matching position area of the template matrix in the prediction matrix according to the "accuracy of each matching position area" as the calculation standard. If the "accuracy of each matching position area" obtained by a certain walking round is 1, the subsequent walking matching is stopped. If the "accuracy of each matching position area" of multiple walking matches is the same and has the highest accuracy, the first walking matching position is selected as the best matching position area. It should be noted that the two horizontal line formalized binary matrices and the two vertical line formalized binary matrices are independent of each other in the process of finding their best matching position areas, and are independently calculated according to the "accuracy of each matching position area".

[0029] In Step 3, for the two horizontal line matrices of the real table structure and the predicted table structure, after finding the best matching position area of the template matrix in the predicted matrix, the element values in the template matrix and the predicted matrix are matched and compared in turn. For the elements in the best matching position area in the predicted matrix, the elements in the template matrix are compared as the true categories in turn. For the elements in the non-best matching position area in the predicted matrix, the absence of horizontal lines is used as the true category for comparison. The true categories of all elements in the template matrix and the predicted categories in the predicted matrix are reflected in the 2×2 confusion matrix ConfusionMatrix4Row. The categories of the confusion matrix ConfusionMatrix4Row are as follows 1; for the two vertical line matrices of the true table structure and the predicted table structure, after finding the best matching position area of the template matrix in the predicted matrix, the values of each element in the template matrix and the predicted matrix are matched and compared in sequence. For the elements in the best matching position area of the predicted matrix, each element in the template matrix is used as the true category for comparison. For the elements in the non-best matching position area of the predicted matrix, the absence of the vertical line is used as the true category for comparison. The true category of all elements in the template matrix and the predicted category in the predicted matrix are reflected in the 2×2 confusion matrix ConfusionMatrix4Col. The categories of the confusion matrix ConfusionMatrix4Col are shown in Table 2 below. Finally, the confusion matrix ConfusionMatrix4Row obtained by matching and comparing the two horizontal line binary matrices and the confusion matrix ConfusionMatrix4Col obtained by matching and comparing the two vertical line binary matrices are added together in sequence to obtain the final 2×2 confusion matrix ConfusionMatrix. The categories of the confusion matrix ConfusionMatrix are shown in Table 3 below.

[0030] Table 1 Confusion Matrix4Row categories obtained after comparing and matching two horizontal line matrices

[0031]

[0032] Table 2 Confusion Matrix4Col categories obtained after comparing and matching the two horizontal line matrices

[0033]

[0034] Table 3 Confusion Matrix categories after the merger

[0035]

[0036] In Step 4, the formulas for calculating objective indicators such as accuracy, precision, recall, and F1-score based on the merged confusion matrix are as follows:

[0037] Accuracy=(TP+TN) / (TP+TN+FP+FN)

[0038] Precision = TP / (TP+FP)

[0039] Recall = TP / (TP+FN)

[0040] F1-score=2*(precision*recall) / (precision+recall).

[0041] While specific embodiments of the present invention have been disclosed for illustrative purposes, intended to facilitate understanding and implementation of the present invention, those skilled in the art will appreciate that various substitutions, variations, and modifications are possible without departing from the spirit and scope of the present invention and the appended claims. Therefore, the present invention should not be limited to the disclosure of the preferred embodiments, and the scope of protection claimed in the present invention shall be determined by the scope of the claims.

Claims

1. A method for evaluating the accuracy of table structure recognition, comprising the following steps: 1) For a table, a markup language is selected to formally represent the table structure of the table, and the table structure before the table is recognized is obtained, which is recorded as the "true table structure"; the table structure prediction model is used to identify the table structure of the table, Recorded as "prediction table structure"; 2) outputting a horizontal line formalized binary matrix and a vertical line formalized binary matrix of the table according to the actual table structure of the table, and outputting a horizontal line formalized binary matrix and a vertical line formalized binary matrix of the table according to the predicted table structure of the table; if a horizontal line exists in a cell of the table, then the element value corresponding to the cell in the horizontal line formalized binary matrix is 1, otherwise it is 0; if a vertical line exists in a cell of the table, then the element value corresponding to the cell in the vertical line formalized binary matrix is 1, otherwise it is 0; 3) using the horizontal line formalized binary matrix and the vertical line formalized binary matrix of the real table structure as the horizontal line template matrix and the vertical line template matrix, respectively, and using the horizontal line formalized binary matrix and the vertical line formalized binary matrix of the predicted table structure as the horizontal line prediction matrix and the vertical line prediction matrix, respectively; Slide the horizontal line template matrix in its corresponding horizontal line prediction matrix with a set step size, traverse the entire horizontal line prediction matrix, and find the best matching position area of the horizontal line template matrix in the corresponding horizontal line prediction matrix; Slide the vertical line template matrix in its corresponding vertical line prediction matrix with a set step size, traverse the entire vertical line prediction matrix, and find the best matching position area of the vertical line template matrix in the corresponding vertical line prediction matrix; 4) When the horizontal line template matrix is located in the best matching position area in the horizontal line prediction matrix, each element in the horizontal line template matrix and the horizontal line prediction matrix is matched and compared to obtain a 2×2 confusion matrix ConfusionMatrix4Row; when the vertical line template matrix is located in the best matching position area in the vertical line prediction matrix, each element in the vertical line template matrix and the vertical line prediction matrix is matched and compared to obtain a 2×2 confusion matrix ConfusionMatrix4Col; then the confusion matrix ConfusionMatrix4Row is merged with the confusion matrix ConfusionMatrix4Col to obtain a confusion matrix ConfusionMatrix; 5) Calculating a table structure recognition accuracy evaluation index based on the confusion matrix ConfusionMatrix to evaluate the table structure recognition accuracy of the table structure prediction model.

2. The method according to claim 1, characterized in that The best matching position area of the template matrix in the prediction matrix is found according to the accuracy of each matching position area; the accuracy of each matching position area = (the number of elements with the same corresponding position element value in each matching position area contained in the template matrix and the prediction matrix / the number of all elements in the template matrix) × 100%; the best matching position area is the matching position area with the highest accuracy; wherein, when the template matrix is a horizontal line template matrix, the corresponding prediction matrix is a horizontal line prediction matrix; when the template matrix is a vertical line template matrix, the corresponding prediction matrix is a vertical line prediction matrix.

3. The method according to claim 2, characterized in that The method for finding the best matching location area is: 31) Align the lower right corner element of the template matrix with the upper left corner element of the prediction matrix; 32) If the size of the submatrix formed by the matching position area of the current template matrix in the prediction matrix is the same as the size of the template matrix, the accuracy of the current matching position area is calculated according to the accuracy of each matching position area; If the submatrix size of the current matching position area is smaller than the template matrix size, the vacant positions in the prediction matrix are filled with elements 0 to obtain the matching position area of the template matrix in the prediction matrix, and then the accuracy of the current matching position area is calculated according to the accuracy of each matching position area; 33) Shift the template matrix to the right by the distance of one element position, and then repeat step 32) until the template matrix traverses the width of the prediction matrix; 34) Move the template matrix downward by the distance of one element position, and then repeat steps 32 to 33) until the template matrix traverses the height of the prediction matrix; 35) The matching position area with the highest accuracy is used as the best matching position area.

4. The method according to claim 1, 2 or 3, characterized in that: The method for obtaining the 2×2 confusion matrix ConfusionMatrix4Row is as follows: when the horizontal line template matrix is located in the best matching position area in the horizontal line prediction matrix, each element in the horizontal line template matrix and the horizontal line prediction matrix is matched and compared. For the elements in the horizontal line prediction matrix in the best matching position area, each element in the horizontal line template matrix is used as the true category for comparison in turn. For the elements in the horizontal line prediction matrix in the non-best matching position area, the absence of horizontal lines is used as the true category for comparison. The true categories of all elements in the horizontal line template matrix and the predicted categories in the horizontal line prediction matrix are reflected in the 2×2 confusion matrix ConfusionMatrix4Row.

5. The method according to claim 1, 2 or 3, characterized in that: The method for obtaining the 2×2 confusion matrix ConfusionMatrix4Col is as follows: when the vertical line template matrix is located in the best matching position area in the vertical line prediction matrix, each element in the vertical line template matrix and the vertical line prediction matrix is matched and compared. For the elements in the vertical line prediction matrix in the best matching position area, each element in the vertical line template matrix is used as the true category for comparison in turn. For the elements in the vertical line prediction matrix in the non-best matching position area, the absence of vertical lines is used as the true category for comparison. The true categories of all elements in the vertical line template matrix and the predicted categories in the vertical line prediction matrix are reflected in the 2×2 confusion matrix ConfusionMatrix4Col.

6. The method according to claim 1, characterized in that The table is a table with m rows and n columns. The method of generating the horizontal line formalized binary matrix and the vertical line formalized binary matrix of the table is: First, define a matrix RM of dimension (m+1)×n and a matrix CM of dimension m×(n+1), where RM represents a horizontal line-formalized binary matrix and CM represents a vertical line-formalized binary matrix. Then assign values to all elements in the matrix RM and the matrix CM respectively to indicate whether the horizontal and vertical lines that make up each cell in the table exist; If there is a complex cell in the table, which is a cell spanning more than one row or column, the complex cell is split into meta-cells, which are indivisible cells spanning only one row and one column; The horizontal or vertical lines used for segmentation are distinguished by dotted lines, and all dotted line positions are mapped one by one to the corresponding positions in RM and CM. The element values at the positions corresponding to the dotted lines are assigned to 0, and the element values at other positions are assigned to 1.

7. The method according to claim 1, characterized in that The table structure recognition accuracy evaluation indicators include Accuracy, Precision, Recall and F1-score.

8. The method according to claim 1, characterized in that The markup language includes HTML language and DOM tree.

9. A server, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, the computer program is configured to be executed by the processor, and the computer program comprises instructions for executing the steps of the method according to any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images