General table line reconstruction method based on object detection

Abstract

A general table line reconstruction method based on object detection, comprising: S1: using a line detection algorithm to perform line detection on actual lines, and then filtering introduced noise; S2: detecting a table object by means of an object detection model to obtain row information, column information, and cell detection information of a table; S3: determining whether there is an overlap by means of the IoU between all row bounding boxes and column bounding boxes, and filtering out overlapping row and column bounding boxes predicted by a model; S4: using the top horizontal line of each row and the left vertical line of each column as reference lines for table alignment, and respectively adding the four borders of a table frame into row and column sets; S5: on the basis of the row and column lines obtained by line detection, aligning lines of all the row bounding boxes and column bounding boxes in the row and column sets to obtain all row and column lines of the table; and S6: on the basis of all the obtained row and column lines of the table in combination with cell bounding boxes and cell attributes, merging cells.

Description

A general table line reconstruction method based on target detection TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, in particular to a general table line reconstruction method based on target detection. BACKGROUND

[0002] Tables are commonly used for data statistics and display. With the development and popularization of computers, electronic document recognition or restoration of tables is commonly used, which requires processing through computer vision technology. However, when restoring, analyzing or reconstructing electronic tables through computers, we not only want to be efficient but also want to ensure the accuracy of the restoration, so higher requirements are put forward for the processing method.

[0003] The invention with publication number CN115620325A discloses a table structure restoration method and device, electronic equipment and storage medium, which reconstructs the table by obtaining the attribute information of the text detection box. However, this method can only obtain the relative position between texts and does not fully utilize the information of table lines in the image. The invention with authorization announcement number CN113221743B discloses a table analysis method and device, electronic equipment and storage medium, which uses table lines and cell information to reconstruct the table and does not process the part across the cells. The invention with publication number CN115797955A discloses a table structure identification method based on cell constraints and its application, which uses a model to predict table row and column information and vertex coordinates, then obtains table structure information and layout information through machine translation, and finally corrects the table and clusters to generate cell structure. However, the whole method is relatively complex and needs to rely on the prediction results of the model, which obviously has efficiency problems.

[0004] In summary, the existing technology usually has the following problems: 1) the original line information or cell information in the image is not fully utilized; 2) the correction effect of the table line is not good and the cross-row cell is not sensitive enough; 3) the processing method is too complex and the model result is too dependent, which has efficiency and accuracy problems. SUMMARY

[0005] To solve the above problems, the purpose of the present application is to provide a general table line reconstruction method based on target detection, which introduces a straight line detection algorithm. Compared with the detection box predicted by the model, the method is more accurate and can correct the output results of the model and align the entire table line. At the same time, the information of the cross-row / cell is used to merge the table lines across the rows or columns, so as to achieve the effect of fine adjustment of table line reconstruction.

[0006] To achieve the above object, the application provides a general table line reconstruction method based on target detection, which comprises the following steps:

[0007] Table line alignment process:

[0008] Step S1: using a straight line detection algorithm to detect the actual lines in the input table image, and filtering the noise introduced by the straight line detection to obtain only the row lines and column lines belonging to the table;

[0009] Step S2: detecting the table targets in the input table image by a target detection model to obtain the row information, column information and cell detection information of the table, and outputting them as the model prediction results, wherein the row information includes row detection boxes, the column information includes column detection boxes, and the cell detection information includes cell detection boxes and cell attributes;

[0010] Step S3: judging whether the IoU between all row detection boxes and column detection boxes in the model prediction results of step S2 is overlapped, and if so, performing threshold filtering to filter out the overlapped row and column detection boxes predicted by the model, and outputting the filtered results as the final prediction results;

[0011] Step S4: taking the top horizontal line of each row and the left vertical line of each column in the final prediction results of step S3 as the cornerstone lines of table alignment, and adding the four edges of the table frame to the row and column sets respectively;

[0012] Step S5: taking the row lines and column lines obtained by the straight line detection in step S1 as the reference to align the lines of all row detection boxes and column detection boxes in the row and column sets:

[0013] If the lines of the row detection boxes and column detection boxes in the final prediction results are within a preset threshold of the row lines and column lines obtained by the straight line detection, the straight line detection results are used as the final line results;

[0014] If the lines of the row detection boxes and column detection boxes in the final prediction results exceed the preset threshold from the row lines and column lines obtained by the straight line detection, both lines are used as the final line results;

[0015] In this way, all the row lines and column lines of the table are obtained;

[0016] Table line merging process:

[0017] Step S6: based on all the row lines and column lines of the table obtained in step S5, combining the cell detection boxes and cell attributes obtained in step S2, merging the corresponding cells, and removing the line segments between the original cells after merging.

[0018] In an embodiment of the present application, the straight line detection algorithm in step S1 uses Hough straight line transformation.

[0019] In an embodiment of the present application, the filtering of noise in step S1 specifically uses morphological operation on the image, and then filtering processing is performed to filter out the non-table horizontal / vertical lines referenced by the straight line detection.

[0020] In an embodiment of the present application, the cell attributes in step S2 include whether it is a title cell, whether it is a sub-title cell, and whether it is a cross-cell.

[0021] In an embodiment of the present application, after step S6, the method further includes:

[0022] Step S7: Based on the results of the straight line detection and in combination with the attributes of the cells, it is judged whether the position of the cell exists across multiple rows or multiple columns, and if it exists, the cross-row line or the cross-column line is removed, that is, the line segment inside the cross-cell is deleted to merge the cells, and finally the correct table line result is obtained.

[0023] The present application provides a general table line reconstruction method based on target detection. Compared with the prior art, since the straight line detection algorithm is introduced, the detection frame obtained by using only the model prediction is more accurate, the model output result can be corrected, and the effect of aligning the entire table line is realized. At the same time, by using the information of the cross-row / cross-column cells, the table lines across rows or columns are merged, and the final effect of the table line reconstruction is finely adjusted. BRIEF DESCRIPTION OF DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0025] FIG. 1 is a flowchart of an embodiment of the present application;

[0026] FIG. 2 is an implementation schematic diagram of table line reconstruction of an embodiment of the present application;

[0027] FIG. 3 is a schematic diagram of row detection results of an embodiment of the present application;

[0028] FIG. 4 is a schematic diagram of column detection results of an embodiment of the present application;

[0029] FIG. 5 is a schematic diagram of row detection frame results of an embodiment of the present application;

[0030] FIG. 6 is a schematic diagram of column detection frame results of an embodiment of the present application;

[0031] Figure 7 is a schematic diagram of the cell detection box result according to an embodiment of the present invention;

[0032] Figure 8 is a schematic diagram of the cell merging result according to an embodiment of the present invention. Detailed Implementation

[0033] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0034] Figure 1 is a flowchart of an embodiment of the present invention. As shown in Figure 1, this embodiment provides a method for general table line reconstruction based on object detection, which includes the following steps:

[0035] Table line alignment process: Using existing object detection models, information including rows, columns, and corresponding attributes is generated. Through normalization and IoU (Intersection over Union) thresholding, ideal rows and columns can be obtained. This process allows for the extraction of corresponding row and column attributes, which can then be used as headers or across cells. Specifically, this includes:

[0036] Step S1: Use a line detection algorithm to detect the actual lines in the input table image, and then filter out the noise introduced by the line detection to obtain only the row and column lines belonging to the table.

[0037] If the input table image is a wireless table, then line detection is not required, and the process proceeds directly to the next step. For wireless tables, the source of the lines in the table is the model detection output of the subsequent step S2. Therefore, wireless tables do not need to go through the alignment process of the subsequent step S5. Thus, the reconstruction method of the present invention is more suitable for wired tables.

[0038] The line detection in this embodiment can be performed using the Hough line transform, but it is not limited to this method.

[0039] Since tables often encounter interference such as small illustrations, colored backgrounds, or closely connected characters, all of which can introduce noise into the line detection results, it is necessary to design more reasonable image morphology operations based on the actual scene and then perform filtering to remove redundant horizontal and vertical lines detected by Hough line transform.

[0040] In this embodiment, the noise filtering in step S1 can specifically involve performing morphological operations on the image and then performing filtering to filter out non-table horizontal / vertical lines referenced by the line detection.

[0041] Step S2: detecting the table target in the input table image by the target detection model to obtain the row information, column information and cell detection information of the table, and output the information as the model prediction result, wherein the row information includes row detection boxes, the column information includes column detection boxes, and the cell detection information includes cell detection boxes (cell text regions) and cell attributes; the target detection model in this embodiment can use an existing model, and the present application is not limited thereto;

[0042] In this embodiment, the cell attributes in step S2 include whether it is a title cell (table header), whether it is a sub-title cell (sub-table header), whether it is a cross-cell (a cell crossing rows or columns), etc.

[0043] Step S3: judging whether the IoU (Intersection over Union, commonly translated as Intersection over Union) between all row detection boxes and column detection boxes in the model prediction result of step S2 is overlapped, and if it is overlapped, performing threshold filtering to filter out the overlapped row and column detection boxes predicted by the model to reduce interference, and outputting the result after filtering as the final prediction result; wherein the filtering can be performed by a rule judgment method;

[0044] The IoU judgment in this embodiment can be performed by calculating the proportion of overlap between each detection box, for example, if the value of IoU is greater than 30%, it is considered that the detection boxes belong to the same region, and one of them is filtered out.

[0045] Step S4: taking the top horizontal line of each row and the left vertical line of each column in the final prediction result of step S3 as the cornerstone line of table alignment, and adding the four edge boxes of the table frame to the row and column sets respectively to ensure that the boundary box of the table region is also added to the set;

[0046] Step S5: using the row lines and column lines detected in step S1 as the reference, aligning the line of all row detection boxes and column detection boxes in the row and column sets:

[0047] If the line of the row detection box and the column detection box in the final prediction result is within a predetermined threshold of the row line and the column line detected by the straight line detection, the straight line detection result is used as the final line result;

[0048] If the line of the row detection box and the column detection box in the final prediction result exceeds the predetermined threshold from the row line and the column line detected by the straight line detection, both lines are used as the final line result;

[0049] In this way, all the row lines and column lines of the table are obtained; by aligning the lines in the table, the correctness of the table line restoration is improved in this embodiment;

[0050] Table line merging process:

[0051] Step S6: based on all the row lines and column lines of the table obtained in step S5, combining the cell bounding box and cell attribute obtained in step S2, merging the corresponding cells, and removing the line segments between the original cells after merging (i.e. the internal line segments of the cells);

[0052] In this embodiment, for example, any cell attribute indicates that it is a cell that needs to be merged, or the cell belongs to the header cell of the table, and these special cells can be merged according to the restoration requirements. After merging, the row and column lines obtained in step S5 will still exist inside the cells. At this time, these lines need to be removed so that there are no extra line segments inside the cells to achieve the restoration of the original table lines.

[0053] After the model prediction result is consistent with the Hough line transformation conversion, a blank row may appear. In order to restore the table lines, the blank row needs to be deleted. Therefore, in this embodiment, step S6 further includes:

[0054] Step S7: based on the results of the straight line detection and combined with the attributes of the cells, it is judged whether the position of the cell exists across multiple rows or multiple columns. If it exists, the cross-row line or cross-column line is removed, that is, the line segment inside the cell is deleted to merge the cell, and finally the correct table line result is obtained.

[0055] Fig. 2 is an implementation schematic diagram of the table line reconstruction of an embodiment of the present application, Fig. 3 is a row detection result schematic diagram of an embodiment of the present application, Fig. 4 is a column detection result schematic diagram of an embodiment of the present application, Fig. 5 is a row detection box result schematic diagram of an embodiment of the present application, Fig. 6 is a column detection box result schematic diagram of an embodiment of the present application, Fig. 7 is a cell detection box result schematic diagram of an embodiment of the present application, and Fig. 8 is a cell merging result schematic diagram of an embodiment of the present application. The embodiments of the present application will be described below with reference to the schematic diagrams shown in Figs. 2-8.

[0056] In order to fully utilize the table line features in the table image, the input table image is subjected to Hough line transformation for straight line detection and noise filtering to obtain row lines and column lines, as shown in the white lines in Figs. 2, 3 and 4;

[0057] The input table image is processed by the target detection model to obtain row detection boxes, column detection boxes and cell detection boxes. Since there are a large number of repeated detection boxes output by the model, the IoU threshold filtering is required to obtain filtered row detection boxes, column detection boxes and cell detection boxes, as shown in Figs. 2, 5, 6 and 7;

[0058] The detection results of the obtained row detection frame, column detection frame, row line and column line are processed by judging the distance between each line to determine whether to be reserved, aligning the original detection result, so as to obtain the processed and aligned table line result.

[0059] The aligned table line is combined by combining the cell detection information, and it is also necessary to judge whether the cell detection frame crosses multiple cells, if it crosses multiple cells, the crossed row line segment or column line segment is removed, for example, the column detection frame in the title row in FIG. 6, the final table line reconstruction result is obtained, as shown in FIG. 8.

[0060] The method for reconstructing general table line based on target detection provided by the application is more accurate than the detection frame obtained by only using model prediction, and can correct the model output result to achieve the effect of aligning the entire table line. Meanwhile, the information of the cross-row / cell is used to combine the table line crossing the row or column, and the final effect of the table line reconstruction is finely adjusted.

[0061] Those skilled in the art can understand that the drawings are only schematic diagrams of an embodiment, and the modules or processes in the drawings are not necessarily necessary for implementing the application.

[0062] Those skilled in the art can understand that the modules in the device in the embodiment can be distributed in the device in the embodiment according to the description of the embodiment, or can be changed and located in one or more devices different from the embodiment. The modules in the above embodiment can be combined into one module, or can be further split into multiple sub-modules.

[0063] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the application, and not to limit them; although the application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the application.

Claims

1. A method for general table line reconstruction based on object detection, characterized in that, The method comprises the following steps: Table line alignment process: Step S1: using a straight line detection algorithm to detect the actual lines in the input table image, and filtering the noise introduced by the straight line detection to obtain only the row lines and column lines belonging to the table; Step S2: detecting the table targets in the input table image through a target detection model to obtain the row information, column information and cell detection information of the table, and outputting the model prediction results, wherein the row information includes row detection boxes, the column information includes column detection boxes, and the cell detection information includes cell detection boxes and cell attributes; Step S3: judging whether the IoU between all row detection boxes and column detection boxes in the model prediction results of step S2 is overlapped, and if so, performing threshold filtering to filter out the overlapped row and column detection boxes predicted by the model, and outputting the filtered results as the final prediction results; Step S4: taking the top horizontal line of each row and the left vertical line of each column in the final prediction results of step S3 as the cornerstone lines of table alignment, and adding the four edges of the table frame to the row and column sets respectively; Step S5: taking the row lines and column lines obtained by the straight line detection in step S1 as the reference to align the lines of all row detection boxes and column detection boxes in the row and column sets: If the lines of the row detection boxes and column detection boxes in the final prediction results are within a preset threshold of the row lines and column lines obtained by the straight line detection, the straight line detection results are used as the final line results; If the lines of the row detection boxes and column detection boxes in the final prediction results exceed the preset threshold from the row lines and column lines obtained by the straight line detection, both lines are used as the final line results; In this way, all the row lines and column lines of the table are obtained; Table line merging process: Step S6: based on all the row lines and column lines of the table obtained in step S5, combining the cell detection boxes and cell attributes obtained in step S2, merging the corresponding cells, and removing the line segments between the original cells after merging.

2. The method of claim 1, wherein, The straight line detection algorithm in step S1 uses Hough straight line transformation.

3. The method for target detection based general table line reconstruction as claimed in claim 1 wherein, In step S1, the noise is filtered by performing morphological operation on the image, and then performing filtering processing to filter out the non-table horizontal / vertical lines introduced by the straight line detection.

4. The method for target detection based general table line reconstruction as claimed in claim 1 wherein, The cell attributes in step S2 include: whether it is a title cell, whether it is a sub-title cell, and whether it is a cross-cell.

5. The method for target detection based general table line reconstruction as claimed in claim 1 wherein, After step S6, it further includes: Step S7: based on the results of the straight line detection and combined with the attributes of the cells, judging whether the position of the cell crosses multiple rows or multiple columns, and if so, removing the cross-row line or cross-column line, i.e. deleting the line segments inside the cross-cell to merge the cells, and finally obtaining the correct table line results.

Images