Automatic reconstruction method of table image based on line detection constraint

Abstract

The invention discloses an automatic reconstruction method of a table image based on line detection constraint. The method comprises the following steps: extracting table line segments in a picture by using a PD-LSD algorithm; merging the obtained line segment endpoints of the table to obtain line segment intersection points of the table; screening the intersection points to obtain representative point coordinates of the cells in the table; building an html table framework by using the obtained representative point coordinates; and converting the built html table into an excel file by utilizing a tablepyxl function library of the python, and outputting the excel file. According to the method, element data information in the file is accurately extracted for the picture containing the table office file by utilizing the PD-LSD algorithm, and the excel table or the html file is output according to the structure of the table, so that the obtained content is accurate and rapid, the office efficiency of a user can be practically improved, the digital transformation of an enterprise and an organization is accelerated, and the user experience is improved. Therefore, the overall working efficiency of enterprises and organizations is better improved.

Description

Technical Field

[0001] This invention belongs to the field of table image processing technology, specifically relating to an automated reconstruction method for table images based on line detection constraints. Background Technology

[0002] In recent years, major enterprises and organizations have been undergoing digital transformation, and converting documents into a computer-readable format is one of the key steps in this process. How to identify the specific structure and content of tables in images and directly extract the data and information for automated table structure reconstruction is a problem of common concern in academia and industry.

[0003] In desktop office settings across various industries, office workers often encounter the need to import table content from images into Excel. Currently, most of these tables are entered manually, which frequently leads to several problems. For example, manual entry often results in tedious secondary reviews due to individual errors, increasing workload. Furthermore, manual entry is highly inefficient when dealing with large volumes of office documents and numerous tables / images requiring conversion. Moreover, with the advancement of information technology, this method is also detrimental to data utilization and analysis.

[0004] In recent years, with the development of technology, the usability of OCR (Optical Character Recognition) has been continuously improved, allowing users to automatically extract text information from images. However, for table scenarios, simply extracting text is not enough. Current OCR technology cannot accurately recognize the specific structure of a table, requiring users to repeatedly copy and paste to reconstruct the spreadsheet, which still consumes a lot of time and cannot avoid errors caused by manual data entry.

[0005] Therefore, there is an urgent need to develop a table extraction, recognition, and reconstruction algorithm that can accurately extract element data information from a file and output it as an Excel spreadsheet or HTML file according to the table structure. Summary of the Invention

[0006] This invention aims to provide an automated reconstruction method for table images based on line detection constraints, to solve the technical problem in existing technologies that struggle to accurately extract table content from images containing office documents. To achieve this objective, the technical solution of this invention is as follows: An automated reconstruction method for table images based on line detection constraints includes the following steps: S1. Use the PD-LSD algorithm to extract table line segments from the image; S2. Merge the endpoints of the table line segments obtained by the PD-LSD algorithm to obtain the intersection points of the table line segments; S3. Filter the intersection points to obtain the coordinates of the representative point of each cell in the table; S4. Use the obtained representative point coordinates to build an HTML table framework; S5. Use Python's tablepyxl library to convert the constructed HTML table into an Excel file for output.

[0007] Furthermore, in step S1, the PD-LSD algorithm further screens the line segment results identified based on the LSD algorithm. It checks whether the ratio of the line segment length to the larger value of the image size's length and the larger value of its width is greater than a certain fixed value. If it is greater than the fixed value, the line segment is retained; otherwise, it is filtered out, so as to obtain the features of line segments that belong only to the table results.

[0008] Specifically, the set value is 0.02~0.05.

[0009] Further, in step S2, the PD-LSD algorithm is used to identify all line segments in the table and output the coordinates and attributes of all line segments. Line segments whose length-to-length ratio is less than 0.02 are then filtered out, thus obtaining all intersection points in the table. Further, in step S3, obtaining the coordinates of the representative points includes: S31. Get the coordinates of the top left corner of a cell: Filter out the intersection of the rightmost column and the bottom row of the table, and the remaining intersection is the top left corner of the corresponding cell; S32. Get the coordinates of the bottom left point of a cell: Use the top left point of each cell to find the point with the closest horizontal coordinate and the smallest vertical coordinate, and get the coordinates of the bottom left point of each cell; S33. Obtain the coordinates of the top right point of a cell: Use the top left point of each cell to find the point with the closest ordinate and the smallest x-coordinate, and obtain the coordinates of the top right point of each cell.

[0010] Furthermore, in step S33, the coordinates of the upper right point of the unit are determined by: finding the midpoint of each line segment and the midpoint between each intersection point and the nearest point in the four directions (up, down, left, and right), taking the intersection of the two, and determining whether the midpoint between each point and the nearest point below is close to a certain point in the intersection. If so, the point is retained; if not, the point is filtered out.

[0011] Furthermore, the step of building the HTML table framework in step S4 includes: S41. After obtaining the coordinates of the top left, bottom left, and top right points of each cell, treat the points whose vertical coordinate difference is less than a certain value as a row and the points whose horizontal coordinate difference is less than a certain value as a column. Count the number of rows and columns, and subtract one from each of the resulting number of rows and columns to get the number of rows and columns of the table. S42. Based on the difference in the horizontal coordinates between the top left and top right points and the difference in the vertical coordinates between the top left and bottom left points, we can determine the size of each cell, that is, the number of columns and rows that each cell spans. S43. Tables can be created in HTML by modifying the colspan and rowspan attributes. colspan is used to specify the number of columns a cell spans horizontally, and rowspan is used to specify the number of rows a cell spans vertically.

[0012] Compared with existing technologies, this invention has the following advantages: For images containing office documents with tables, the PD-LSD algorithm is used to accurately extract element data information from the file and output it as an Excel spreadsheet or HTML file according to the table structure. The content acquisition is accurate and fast, which can effectively improve users' office efficiency, accelerate the digital transformation of enterprises and organizations, and thus better improve the overall work efficiency of enterprises and organizations. Attached Figure Description

[0013] Figure 1 This is a flowchart of the method of the present invention; Figure 2 This is an example diagram of the merged intersection points in an embodiment of the present invention; Figure 3 This is an example diagram of redundant intersection points in an embodiment of the present invention; Figure 4 This is an example diagram illustrating the removal of redundant intersections in an embodiment of the present invention; Figure 5 This is an example diagram of an HTML table in an embodiment of the present invention; Figure 6 This is an example image showing the result of exporting an Excel file in an embodiment of the present invention; Figure 7 The results of existing LSD algorithm recognition and local magnification; Figure 8 The image shows the recognition result and a magnified view of the PD-LSD algorithm of this invention. Figure 9 The image shows the recognition results of the existing YOLO5 algorithm (the circled cells are those missed by the YOLO5 algorithm). Figure 10 This is a diagram showing the recognition results of the PD-LSD algorithm of this invention; Figure 11 This is a comparison chart of the recognition results between the existing YOLO5 algorithm and the PD-LSD algorithm of this invention. Detailed Implementation

[0014] The following description, in conjunction with the accompanying drawings, further illustrates an automated reconstruction method for table images based on line detection constraints according to the present invention.

[0015] like Figure 1 As shown, the automated reconstruction method for table images based on line detection constraints includes the following steps: Step 1: Extract table line segments from the image using the PD-LSD algorithm. Specifically, the PD-LSD algorithm further filters the line segments identified by the LSD algorithm. It checks if the ratio of the line segment length to the larger of the image's length and width is greater than a certain fixed value. This fixed value is between 0.02 and 0.05 (tested, 0.02 is generally more common; actual application may vary depending on the table). If the ratio is greater than this fixed value, the line segment is retained; otherwise, it is discarded. This yields features unique to line segments belonging to the table results.

[0016] Step two involves extracting the final table structure based on the line segment content identified by the PD-LSD algorithm, using appropriate strategies. The specific strategies are as follows: (1) Merge the endpoints of the line segments obtained by the PD-LSD algorithm to obtain the intersection points of the line segments in the table; the PD-LSD line detection algorithm identifies all line segments in the table and outputs the coordinates and attributes of all line segments, such as the start point, end point, and line thickness. Then, we merge the coordinates of the endpoints whose distance accounts for a larger proportion of the photo length than a certain value to obtain all intersection points in the table, such as... Figure 2 The diagram shows an example of a merged intersection point.

[0017] By filtering the intersection points, the coordinates of the representative point in each cell of the table are obtained.

[0018] First, obtain the coordinates of the top-left point of the cell. After filtering out the intersection points of the rightmost column and the bottom row of the table, most of the remaining intersection points are the top-left points of the corresponding cells.

[0019] Secondly, obtain the coordinates of the bottom left point of the cell: use the top left point of each cell to find the point with the closest horizontal coordinate and the smallest vertical coordinate, and obtain the coordinates of the bottom left point of each cell; Finally, obtain the coordinates of the top-right point of each cell: Find the point with the closest ordinate and the smallest x-coordinate distance between the top-left points of each cell to obtain the coordinates of the top-right point for each cell. When calculating the top-right point of each cell, because some cells are merged and span a long distance, there may be extra intersection points between the top-left and top-right points, such as... Figure 3 The extra intersection points marked can interfere with the program's judgment of the top-right point of a cell, leading to incorrect cell lengths. Therefore, this project calculates the midpoint of each line segment and the midpoints of each intersection point with the nearest points in each of the four cardinal directions (up, down, left, and right), and takes the intersection of these two sets. Finally, it checks if the midpoint between each point and its nearest lower point is close to any point in the intersection set. If so, the point is retained; otherwise, it is discarded. The results are as follows: Figure 4The example image shown shows the result after filtering out redundant intersections. It is clear from the image that the redundant points have been filtered out. By using the top left point of each cell to find the point with the closest vertical coordinate and the smallest horizontal coordinate, it is easy to obtain the coordinates of the top right point of each cell.

[0020] (3) Using the obtained representative point coordinates, construct the HTML table framework. Specifically, after obtaining the coordinates of the top-left, bottom-left, and top-right points of each cell, treat points with a vertical coordinate difference less than a certain value as a row and points with a horizontal coordinate difference less than a certain value as a column, and count the number of rows and columns. Subtract one from each of the obtained row and column numbers to obtain the number of rows and columns of the table. In addition, based on the horizontal coordinate difference between the top-left and top-right points and the vertical coordinate difference between the top-left and bottom-left points, the size of each cell can be determined, i.e., the number of columns and rows that each cell spans. Finally, create the table in the HTML by modifying the `colspan` and `rowspan` attributes, where `colspan` is used to specify the number of columns a cell spans horizontally, and `rowspan` is used to specify the number of rows a cell spans vertically. Figure 5 The image shown is an example of an HTML table. The "111" content in the image is for better display of the table style and has no special meaning.

[0021] (4) Use Python's tablepyxl library to convert the constructed HTML table into an Excel file for output. For example... Figure 6 The image shown is an example of the results from the Excel file.

[0022] After the above steps, better results can be obtained than those of mainstream table reconstruction algorithms on the market.

[0023] As can be seen from the comparison chart of the PD-LSD algorithm and the original LSD algorithm, the present invention can obtain more accurate table content. Figure 7 The results of the existing LSD algorithm recognition and a magnified view of a local area are shown. Figure 8 The image shown is the recognition result and a magnified view of the PD-LSD algorithm of this invention.

[0024] A comparison of the specific performance of the method of this invention with mainstream target bounding box detection algorithms on the market (taking YOLOv5 as an example) Figures 9-11 It can be seen that the results obtained by the present invention are more accurate. Figure 9 The image shows the recognition results of the existing YOLO5 algorithm (the circled cells are those missed by the YOLO5 algorithm). Figure 10 This is a diagram showing the recognition result of the PD-LSD algorithm of this invention. Figure 11The image shows a comparison of the recognition results of the YOLO5 algorithm and the PD-LSD algorithm (Note: The left image shows the recognition result of the YOLO5 algorithm, and the circled area shows the extra cells incorrectly identified by the YOLO5 algorithm; the right image shows the result obtained by the PD-LSD algorithm).

[0025] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An automated reconstruction method for table images based on line detection constraints, characterized in that, The method includes the following steps: S1. Use the PD-LSD algorithm to extract table line segments from the image; S2. Merge the endpoints of the table line segments obtained by the PD-LSD algorithm to obtain the intersection points of the table line segments; S3. Filter the intersection points to obtain the coordinates of the representative point of each cell in the table; S4. Use the obtained representative point coordinates to build an HTML table framework; S5. Use Python's tablepyxl library to convert the constructed HTML table into an Excel file for output.

2. The automated reconstruction method for table images based on line detection constraints according to claim 1, characterized in that, In step S1, the PD-LSD algorithm further screens the line segment results identified by the LSD algorithm. It checks whether the ratio of the larger value of the line segment length to the larger value of the image size length and the larger value of the width is greater than a certain fixed value. If it is greater than the fixed value, the line segment is retained; otherwise, it is filtered out, so as to obtain the features of line segments that belong only to the table results.

3. The automated reconstruction method for table images based on line detection constraints according to claim 2, characterized in that, The set value is 0.02~0.

05.

4. The automated reconstruction method for table images based on line detection constraints according to claim 3, characterized in that, In step S2, the PD-LSD algorithm is used to identify all line segments in the table and output the coordinates and attributes of all line segments. After filtering out line segments whose length ratio to the length of the longer side of the image is less than 0.02, all intersection points in the table can be obtained.

5. The automated reconstruction method for table images based on line detection constraints according to claim 1, characterized in that, The acquisition of the point coordinates in step S3 includes: S31. Get the coordinates of the top left corner of a cell: Filter out the intersection of the rightmost column and the bottom row of the table, and the remaining intersection is the top left corner of the corresponding cell; S32. Get the coordinates of the bottom left point of a cell: Use the top left point of each cell to find the point with the closest horizontal coordinate and the smallest vertical coordinate, and get the coordinates of the bottom left point of each cell; S33. Obtain the coordinates of the top right point of a cell: Use the top left point of each cell to find the point with the closest ordinate and the smallest x-coordinate, and obtain the coordinates of the top right point of each cell.

6. The automated reconstruction method for table images based on line detection constraints according to claim 5, characterized in that, In step S33, the coordinates of the upper right point of the unit are determined as follows: the midpoint of each line segment and the midpoint between each intersection point and the nearest point in the four directions (up, down, left, and right) are obtained. The intersection of the two is taken, and it is determined whether the midpoint between each point and the nearest point below is close to a point in the intersection. If so, the point is retained; otherwise, the point is discarded.

7. The automated reconstruction method for table images based on line detection constraints according to claim 1, characterized in that, The step of building the HTML table framework in step S4 includes: S41. After obtaining the coordinates of the top left, bottom left, and top right points of each cell, treat the points whose vertical coordinate difference is less than a certain value as a row and the points whose horizontal coordinate difference is less than a certain value as a column. Count the number of rows and columns, and subtract one from each of the resulting number of rows and columns to get the number of rows and columns of the table. S42. Based on the difference in the horizontal coordinates between the top left and top right points and the difference in the vertical coordinates between the top left and bottom left points, we can determine the size of each cell, that is, the number of columns and rows that each cell spans. S43. Tables can be created in HTML by modifying the colspan and rowspan attributes. colspan is used to specify the number of columns a cell spans horizontally, and rowspan is used to specify the number of rows a cell spans vertically.

Images