General table line reconstruction method based on target detection

Abstract

A general table line reconstruction method based on target detection comprises the following steps: S1, carrying out straight line detection on an actual line by using a straight line detection algorithm, and filtering introduced noise; s2, detecting a table target through a target detection model to obtain row information and column information of the table and detection information of cells; s3, judging whether the row detection frames and the column detection frames are overlapped or not through IoU between all the row detection frames and all the column detection frames, and filtering the overlapped row detection frames and the column detection frames predicted by the model; s4, taking the top horizontal line of each row and the left vertical line of each column as footstone lines for table alignment, and respectively adding four borders of the table frame into the row and column sets; s5, aligning lines of all row detection frames and all column detection frames in the row and column set by taking the row lines and the column lines obtained by straight line detection as a reference to obtain all row lines and column lines of the table; and S6, combining the cells based on all the row lines and the column lines of the obtained table in combination with the cell detection frame and the cell attributes.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and more specifically, to a method for reconstructing general table lines based on object detection. Background Technology

[0002] Tables are a common way for us to collect and display data. With the development and popularization of computers, we often use electronic documents to recognize or restore tables. This requires processing through computer vision technology. However, when restoring, parsing or reconstructing electronic tables using computers, we want both efficiency and accuracy. Therefore, we have higher requirements for processing methods.

[0003] Invention CN115620325A discloses a method, apparatus, electronic device, and storage medium for restoring table structures. It reconstructs the table by acquiring the attribute information of text detection boxes; however, this only obtains the relative positions between text elements and does not fully utilize the information of table lines in the image. Invention CN113221743B discloses a table parsing method, apparatus, electronic device, and storage medium, which reconstructs the table using only table lines and cell information, without processing cross-cell portions. Invention CN115797955A discloses a table structure recognition method based on cell constraints and its application. It uses a model to predict table row and column information and vertex coordinates, then obtains table structure and layout information through machine translation, and then corrects the table and clusters it to generate cell structures. However, the entire method is complex and relies heavily on model prediction results, clearly exhibiting efficiency drawbacks.

[0004] In summary, existing technologies typically have the following problems: 1) They do not fully utilize the original line or cell information of the image; 2) They are not effective in correcting table lines and are not sensitive enough to cells spanning multiple rows; 3) Overly complex processing methods and excessive reliance on model results lead to efficiency and accuracy issues. Summary of the Invention

[0005] To address the aforementioned problems, the present invention aims to provide a general table line reconstruction method based on object detection. This method introduces a line detection algorithm, which is more accurate than detection boxes predicted solely by the model, and can correct model output results and align the entire table line. Furthermore, by utilizing information from cells spanning multiple rows / columns, table lines spanning multiple rows or columns are merged to achieve a finely adjusted table line reconstruction effect.

[0006] To achieve the above objectives, this invention provides a method for general table line reconstruction based on object detection, which includes the following steps:

[0007] Table line alignment process:

[0008] Step S1: Use a line detection algorithm to detect the actual lines in the input table image, and then filter out the noise introduced by the line detection to obtain only the row and column lines belonging to the table.

[0009] Step S2: Detect table targets in the input table image using the object detection model to obtain the table's row information, column information, and cell detection information, and output them as the model's prediction results. The row information includes row detection boxes, the column information includes column detection boxes, and the cell detection information includes cell detection boxes and cell attributes.

[0010] Step S3: Determine whether there is overlap between all row and column detection boxes in the model prediction results of step S2. If there is overlap, perform threshold filtering to filter out the overlapping row and column detection boxes predicted by the model, and output the filtered result as the final prediction result.

[0011] Step S4: Using the top horizontal line of each row and the left vertical line of each column in the final prediction result of step S3 as the base line for table alignment, add the four borders of the table frame to the row and column sets respectively.

[0012] Step S5: Using the row and column lines obtained from the line detection in Step S1 as a reference, align the lines of all row and column detection boxes in the row and column sets:

[0013] If the row and column lines detected by the line detection boxes in the final prediction result are within a preset threshold, then the line detection result is used as the final line result.

[0014] If the line between the row and column detection boxes in the final prediction result exceeds the preset threshold, then both lines will be used as the final line result.

[0015] This will give you all the row and column lines of the table;

[0016] Table line merging process:

[0017] Step S6: Based on all the row and column lines of the table obtained in Step S5, and combined with the cell detection boxes and cell attributes obtained in Step S2, merge the corresponding cells and remove the line segments between the original cells after merging.

[0018] In one embodiment of the present invention, the line detection algorithm in step S1 employs the Hough line transform.

[0019] In one embodiment of the present invention, step S1 of filtering noise specifically involves performing morphological operations on the image and then performing filtering processing to filter out non-table horizontal / vertical lines referenced by the line detection.

[0020] In one embodiment of the present invention, the cell attributes in step S2 include: whether it is a title cell, whether it is a subtitle cell, and whether it spans multiple cells.

[0021] In one embodiment of the present invention, step S6 is followed by:

[0022] Step S7: Based on the results of the line detection and the cell attributes, determine whether the cell position spans multiple rows or columns. If so, remove the line crossings, that is, delete the line segments that cross the inside of the cell to merge the cells and finally get the correct table line result.

[0023] This invention provides a general table line reconstruction method based on object detection. Compared with existing technologies, it introduces a line detection algorithm, which is more accurate than the detection boxes obtained by using only model prediction. This method can correct the model output and achieve the effect of aligning the entire table line. At the same time, it utilizes information from cells spanning multiple rows / columns to merge table lines that span multiple rows or columns, finely adjusting the final effect of the table line reconstruction. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 This is a flowchart illustrating an embodiment of the present invention;

[0026] Figure 2 This is a schematic diagram illustrating an embodiment of table line reconstruction according to the present invention;

[0027] Figure 3 This is a schematic diagram of the row detection results according to an embodiment of the present invention;

[0028] Figure 4 This is a schematic diagram of column detection results according to an embodiment of the present invention;

[0029] Figure 5 This is a schematic diagram of the row detection box result according to an embodiment of the present invention;

[0030] Figure 6 This is a schematic diagram of the column detection box results according to an embodiment of the present invention;

[0031] Figure 7 This is a schematic diagram of the cell detection box result according to an embodiment of the present invention;

[0032] Figure 8 This is a schematic diagram of the cell merging result according to an embodiment of the present invention. Detailed Implementation

[0033] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0034] Figure 1 This is a flowchart illustrating an embodiment of the present invention, as shown below. Figure 1 As shown, this embodiment provides a method for general table line reconstruction based on object detection, which includes the following steps:

[0035] Table line alignment process: Using existing object detection models, information including rows, columns, and corresponding attributes is generated. Through normalization and IoU (Intersection over Union) thresholding, ideal rows and columns can be obtained. This process allows for the extraction of corresponding row and column attributes, which can then be used as headers or across cells. Specifically, this includes:

[0036] Step S1: Use a line detection algorithm to detect the actual lines in the input table image, and then filter out the noise introduced by the line detection to obtain only the row and column lines belonging to the table.

[0037] If the input table image is a wireless table, then line detection is not required, and the process proceeds directly to the next step. For wireless tables, the source of the lines in the table is the model detection output of the subsequent step S2. Therefore, wireless tables do not need to go through the alignment process of the subsequent step S5. Thus, the reconstruction method of the present invention is more suitable for wired tables.

[0038] The line detection in this embodiment can be performed using the Hough line transform, but it is not limited to this method.

[0039] Since tables often encounter interference such as small illustrations, colored backgrounds, or closely connected characters, all of which can introduce noise into the line detection results, it is necessary to design more reasonable image morphology operations based on the actual scene and then perform filtering to remove redundant horizontal and vertical lines detected by Hough line transform.

[0040] In this embodiment, the noise filtering in step S1 can specifically involve performing morphological operations on the image and then filtering it to remove non-table horizontal / vertical lines referenced by the line detection.

[0041] Step S2: The target detection model is used to detect the table targets in the input table image, and the table row information, column information and cell detection information are obtained. These are then output as the model prediction results. The row information includes the row detection box, the column information includes the column detection box, and the cell detection information includes the cell detection box (cell text area) and cell attributes. The target detection model in this embodiment can use existing models, but this invention is not limited thereto.

[0042] In this embodiment, the cell attributes in step S2 include: whether it is a header cell, whether it is a subheading cell, and whether it spans multiple cells (a cell that spans multiple rows or columns).

[0043] Step S3: Determine whether there is overlap between all row and column detection boxes in the model prediction results of step S2 by using the IoU (Intersection over Union). If there is overlap, perform threshold filtering to filter out the overlapping row and column detection boxes predicted by the model to reduce interference, and output the filtered result as the final prediction result. The filtering can be performed using rule-based judgment methods.

[0044] In this embodiment, the IoU determination can be achieved by calculating the overlap ratio between each detection box. For example, if the calculated IoU value is greater than 30%, the detection boxes are considered to belong to the same region, and one of them can be filtered out.

[0045] Step S4: Using the top horizontal line of each row and the left vertical line of each column in the final prediction result of step S3 as the base line for table alignment, add the four borders of the table frame to the row and column sets respectively to ensure that the bounding box of the table area is also added to the set.

[0046] Step S5: Using the row and column lines obtained from the line detection in Step S1 as a reference, align the lines of all row and column detection boxes in the row and column sets:

[0047] If the row and column lines detected by the line detection boxes in the final prediction result are within a preset threshold, then the line detection result is used as the final line result.

[0048] If the line between the row and column detection boxes in the final prediction result exceeds the preset threshold, then both lines will be used as the final line result.

[0049] This yields all row and column lines of the table; this embodiment improves the accuracy of table line restoration by aligning the lines in the table.

[0050] Table line merging process:

[0051] Step S6: Based on all the row and column lines of the table obtained in Step S5, and combined with the cell detection box and cell attributes obtained in Step S2, merge the corresponding cells and remove the line segments between the original cells after merging (i.e., the line segments inside the cells).

[0052] In this embodiment, for example, if any cell attribute shows that it is a cell that needs to be merged, or if the cell belongs to the table header, these special cells can be merged according to the restoration requirements. After merging, the cell will still contain the row and column lines obtained in step S5. At this time, these lines need to be removed so that there are no more extra lines inside the cell, so as to restore the original table lines.

[0053] After the model prediction results are consistent with the Hough linear transformation, blank rows may appear. To restore the table lines, the blank rows need to be deleted. Therefore, in this embodiment, step S6 is followed by:

[0054] Step S7: Based on the results of the line detection and the cell attributes, determine whether the cell position spans multiple rows or columns. If so, remove the line crossings, that is, delete the line segments that cross the inside of the cell to merge the cells and finally get the correct table line result.

[0055] Figure 2 This is a schematic diagram illustrating an embodiment of table line reconstruction according to the present invention. Figure 3 This is a schematic diagram of the row detection results according to an embodiment of the present invention. Figure 4 This is a schematic diagram of column detection results according to an embodiment of the present invention. Figure 5 This is a schematic diagram of the row detection box result according to an embodiment of the present invention. Figure 6 This is a schematic diagram of the column detection box results according to an embodiment of the present invention. Figure 7 This is a schematic diagram of the cell detection box result according to an embodiment of the present invention. Figure 8 This is a schematic diagram of the cell merging result according to an embodiment of the present invention. The following will use... Figures 2-8 The schematic diagrams shown illustrate embodiments of the present invention.

[0056] To fully utilize the table line features in the table image, the input table image undergoes Hough line transform to detect lines and filter out noise, resulting in row and column lines, as shown below. Figure 2 , Figure 3 and Figure 4 The white lines in the middle;

[0057] The input table image is processed by the object detection model to obtain row detection boxes, column detection boxes, and cell detection boxes. Since the model outputs a large number of duplicate detection boxes, they also need to be filtered using an IoU threshold to obtain filtered row detection boxes, column detection boxes, and cell detection boxes, such as... Figure 2 , Figure 5 , Figure 6 and Figure 7 As shown;

[0058] The detection results of the row detection boxes, column detection boxes, row lines, and column lines obtained above are used to determine whether to retain them by judging the distance between each line, and the original detection results are aligned to obtain the processed and aligned table line results.

[0059] Based on cell detection information, the aligned table lines are merged. This process also requires determining if the cell detection box spans multiple cells; if so, the crossed row or column segments are removed. For example... Figure 6 The middle column detection box is located on the column line segment of the header row, resulting in the final table line reconstruction result, such as... Figure 8 As shown.

[0060] This invention provides a general table line reconstruction method based on object detection. By introducing a line detection algorithm, the method achieves higher accuracy compared to detection boxes obtained solely using model prediction, correcting model output and aligning the entire table line. Furthermore, it utilizes information from cells spanning multiple rows or columns to merge table lines that cross rows or columns, finely adjusting the final table line reconstruction result.

[0061] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of one embodiment, and the modules or processes shown in the drawings are not necessarily essential for implementing the present invention.

[0062] Those skilled in the art will understand that the modules in the apparatus of the embodiments can be distributed in the apparatus of the embodiments as described in the embodiments, or they can be located in one or more devices different from this embodiment with corresponding changes. The modules of the above embodiments can be combined into one module, or they can be further divided into multiple sub-modules.

[0063] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for general table line reconstruction based on object detection, characterized in that, Includes the following steps: Table line alignment process: Step S1: Use a line detection algorithm to detect the actual lines in the input table image, and then filter out the noise introduced by the line detection to obtain only the row and column lines belonging to the table. Step S2: Detect table targets in the input table image using the object detection model to obtain the table's row information, column information, and cell detection information, and output them as the model's prediction results. The row information includes row detection boxes, the column information includes column detection boxes, and the cell detection information includes cell detection boxes and cell attributes. Step S3: Determine whether there is overlap between all row and column detection boxes in the model prediction results of step S2. If there is overlap, perform threshold filtering to filter out the overlapping row and column detection boxes predicted by the model, and output the filtered result as the final prediction result. Step S4: Using the top horizontal line of each row and the left vertical line of each column in the final prediction result of step S3 as the base line for table alignment, add the four borders of the table frame to the row and column sets respectively. Step S5: Using the row and column lines obtained from the line detection in Step S1 as a reference, align the lines of all row and column detection boxes in the row and column sets: If the row and column lines detected by the line detection boxes in the final prediction result are within a preset threshold, then the line detection result is used as the final line result. If the line between the row and column detection boxes in the final prediction result exceeds the preset threshold, then both lines will be used as the final line result. This will give you all the row and column lines of the table; Table line merging process: Step S6: Based on all the row and column lines of the table obtained in Step S5, and combined with the cell detection boxes and cell attributes obtained in Step S2, merge the corresponding cells and remove the line segments between the original cells after merging.

2. The method for general table line reconstruction based on object detection according to claim 1, characterized in that, The line detection algorithm in step S1 uses the Hough line transform.

3. The method for general table line reconstruction based on object detection according to claim 1, characterized in that, Step S1 filters noise by performing morphological operations on the image and then filtering it to remove non-table horizontal / vertical lines referenced by the line detection.

4. The method for general table line reconstruction based on object detection according to claim 1, characterized in that, The cell attributes in step S2 include: whether it is a header cell, whether it is a subheading cell, and whether it spans multiple cells.

5. The method for general table line reconstruction based on object detection according to claim 1, characterized in that, Step S6 is followed by: Step S7: Based on the results of the line detection and the cell attributes, determine whether the cell position spans multiple rows or columns. If so, remove the line crossings, that is, delete the line segments that cross the inside of the cell to merge the cells and finally get the correct table line result.

Images