Table image structuring information extraction and analysis method and device based on character detection

Abstract

This invention discloses a method and apparatus for extracting and analyzing structured information from table images based on text detection. The method includes: performing text detection on the table image to obtain the position information of each text detection box; performing preliminary row division based on the position information of the text detection boxes to obtain preliminary row division results; selecting multiple reference rows from the preliminary row division results and constructing reference row groups; re-dividing the preliminary row division according to the reference row groups to obtain re-row division results; determining column types based on the re-row division results, and grouping and columnizing according to column types. This method and apparatus obtain the row and column information of the table based on the position information of the text detection boxes, thereby realizing the extraction and analysis of structured information. This method and apparatus require no manual annotation, have low computational load, and require no additional hardware resources.

Description

Technical Field

[0001] This invention belongs to the field of structured information extraction technology, specifically relating to a method and apparatus for extracting and analyzing structured information from table images based on text detection. Background Technology

[0002] The application scenarios for table-based image structured information extraction and analysis are wide-ranging. For example, it can convert tables in paper documents, electronic documents, or web pages into editable electronic formats, facilitating data storage, retrieval, and analysis. It can also be used to extract key information from complex tables, supporting decision-making and knowledge discovery.

[0003] Typically, the extraction and analysis of structured information from table images involves two sub-tasks: table detection and table structure recognition. Table detection refers to locating table regions in a document image, while table structure recognition involves reconstructing the row and column information of the table, as well as the coordinates and content of each cell. Currently, several deep learning-based methods have been proposed, such as the GFTE method based on Graph Convolutional Networks (GCN), the TIES method based on Fully Convolutional Networks (FCN) and Conditional Generative Adversarial Networks (CGAN), and the TSRFormer method based on Transformers. These methods have been tested on some publicly available datasets and have achieved certain results. However, some problems and challenges remain, such as the size and quality of the dataset. Typically, this method requires the prior collection and labeling of large datasets, which consumes significant human resources. Furthermore, this method usually requires substantial computational resources.

[0004] Patent document CN112883795A discloses a method for fast and automatic table extraction based on deep neural networks, including: performing image table correction preprocessing; extracting image table outlines to obtain cropped cell images; locating cell text; recognizing cell text; and sequentially merging all cell recognition results to achieve automatic and fast table extraction.

[0005] Patent application CN113239818A discloses a method for extracting cross-modal information from table images based on image segmentation and graph convolutional neural networks. The method includes: using a deep learning object detection method to obtain the coordinates of the corner points of each node in the table, and using the obtained corner point coordinates and an OCR interface to obtain the text information in each node of the table; using an image segmentation model to divide the table into functional regions—the header region, attribute region, data region, and top-left corner region—based on the features of the table image; for the nodes in the header and attribute regions, using the text, coordinates, and image multimodal information features of each node, inferring the edge relationships between nodes through a graph convolutional deep model to extract the topological relationships between table nodes; reconstructing the graph model structure of the header and attribute regions based on the topological relationships; obtaining the number of rows and columns in the data region based on the number of nodes in the lowest layer of the graph structure of the header and attribute regions, and using the data region nodes to fill the table data region; and reconstructing the structure of the entire table based on the node graph structure of the header and attribute regions and the reconstruction results of the table region.

[0006] Both of the above technical solutions require the model to be trained with a large amount of sample data before it can be applied, which will cause technical problems such as high manpower consumption and high computational cost. Summary of the Invention

[0007] In view of the above, the purpose of this invention is to provide a method and apparatus for extracting and analyzing structured information from table images based on text detection. The method and apparatus obtain the row and column information of the table based on the position information of the text detection box, thereby realizing the extraction and analysis of structured information. The method and apparatus do not require manual annotation, have low computational load, and do not require additional hardware resources.

[0008] To achieve the aforementioned objectives, an embodiment provides a method for extracting and analyzing structured information from table images based on text detection, comprising:

[0009] Compared with the prior art, the beneficial effects of the present invention include at least the following:

[0010] The method and apparatus for extracting and analyzing structured information from table images based on text detection provided by this invention do not directly detect and analyze table lines. Instead, they directly obtain the row and column information of the text using the position information of the text detection boxes. This method and apparatus have good adaptability to curved images and are suitable for the extraction and analysis of structured information from most table images. This invention requires no manual annotation and has a very small computational load, making it suitable for most computing devices and requiring no additional hardware resources. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a flowchart of the method for extracting and analyzing structured information from table images based on text detection, provided in the embodiment.

[0013] Figure 2 This is a schematic diagram of the structure of the table image structured information extraction and analysis device based on text detection provided in the embodiment. Detailed Implementation

[0014] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not limit the scope of protection of this invention.

[0015] The inventive concept of this invention is to solve the technical problem of wasting computational and human resources in the extraction of structured information from table images. The embodiment provides a method and device for extracting and analyzing structured information from table images based on text detection. The row and column information of the table is obtained based on the position information of the text detection box, thereby realizing the extraction and analysis of structured information. It does not require manual annotation, has a small amount of computation, and does not require additional hardware resources.

[0016] Figure 1 This is a flowchart of a method for extracting and analyzing structured information from table images based on text detection, provided in an embodiment. For example... Figure 1 As shown in the embodiment, the method for extracting and analyzing structured information from table images based on text detection includes the following steps:

[0017] S110, perform text detection on the table image to obtain the position information of each text detection box, perform preliminary row division based on the position information of the text detection boxes, and obtain preliminary row division results.

[0018] In this embodiment, text detection is performed on the table image to obtain the position information of each text detection box. The position information can be represented by an outer (rotated) rectangle box. i (x,y,w,h,a) means that x is the x-coordinate of the top-left corner, y is the y-coordinate of the top-left corner, w is the width of the frame, h is the height of the frame, and a is the rotation angle. If it is a positive frame, the rotation angle is 0.

[0019] In this embodiment, preliminary line breaks based on the position information of the text detection boxes include: sorting all text detection boxes into a list according to their horizontal coordinates; traversing the sorted list; matching the current text detection box with an existing preliminary line break list; if a match is found, storing the current text detection box in the existing preliminary line break list; otherwise, generating a new preliminary line break list for the current text detection box and storing it.

[0020] The process of matching the current text detection box with the existing preliminary line break list includes: determining whether the current text detection box and the last text detection box in the existing preliminary line break list belong to the same line based on the relative position of the two text detection boxes; if so, using the preliminary existing line break list containing the last text detection box as the current text detection box as the candidate line break list; if there is only one candidate line break list corresponding to the current text detection box, then that candidate line break list is considered to be the existing preliminary line break list that the current text detection box matches; if there are multiple candidate line break lists corresponding to the current text detection box, then selecting the candidate line break list with the closest relative position as the existing preliminary line break list that the current text detection box matches.

[0021] In this embodiment, the judgment based on the relative position of two text detection boxes includes: if the horizontal coordinates of the two text detection boxes overlap, or the angle between the two text detection boxes is greater than a threshold, then the two text detection boxes are considered not to belong to the same row; otherwise, they are considered to belong to the same row; if the angle between the two text detection boxes is the smallest, or the angle change is the smallest, then the relative position of the two text detection boxes is considered to be the smallest.

[0022] Specifically, let the initial row break result be a list named rows, and each element in the initial row break list rows is row. k This is a list of text detection boxes, indicating that each detection box belongs to a line from left to right. The list {box0, box1, ..., box...} is sorted by its x-coordinate. i If we set `}`, then `box0` must be at the initial position of a row, so we initialize the list `rows` to `[[box0]`.

[0023] For each subsequent text detection box in the sorted list i Iterate through the last text checkbox of each row in the list rows. j Matching and judging boxes i Is it related to box? j They belong to the same row. There are several ways to make this judgment. For example, a more lenient approach is to assume that as long as the two text detection boxes are definitely not on the same row (e.g., they overlap in the horizontal direction or the angle between the two boxes is greater than a certain angle), the two text detection boxes are temporarily considered to belong to the same row.

[0024] For box i If no matching is found in the existing preliminary row list using the above matching method, then the box is considered to be missing. i At the beginning of a new line, if one or more lines are matched, the most suitable line needs to be selected. There can be various selection rules, such as selecting the line with the smallest angle between two document detection boxes or the line with the smallest angle change as the matching line. Then, the box... i Add it to the end of the initial line break list. This process is repeated until all text detection boxes have been traversed, thus completing the initial line break.

[0025] S120: Select multiple reference rows from the initial row splitting results and construct a reference row group. Re-split the initial row splitting based on the reference row group to obtain the re-split results.

[0026] Since the initial line splitting condition for determining whether two text detection boxes belong to the same line is lenient, and only a portion of the text detection boxes are seen during the initial line splitting, a reference line needs to be selected from the initial line splitting results to obtain a more accurate line splitting result. The criteria for selecting the reference line can be varied. Generally, it is desirable for the reference line to meet the following conditions: (1) it contains a large number of text detection boxes; (2) the transition between the angles of the text detection boxes is relatively smooth.

[0027] If there is only one row as a reference row, then the reference significance for text detection boxes that are farther away is relatively small. Therefore, it is desirable to have more rows as reference rows to form a reference row group. The general principle for selecting a reference row group is as follows: if the current row and a certain reference row have similar coverage on the horizontal axis (to avoid "too short" rows lacking reference significance), and the distance between each text detection box and the nearest text detection box in the reference row is similar (such as not exceeding 0.5 times the row height), then the current row can be added to the reference row group as a reference row.

[0028] Based on the above analysis requirements, in this embodiment, multiple reference rows are selected from the preliminary row division results and a reference row group is constructed. This includes: selecting reference rows from the preliminary row division results and adding them to the reference row group based on the condition that the number of text detection boxes in the preliminary row division list reaches a threshold and the angle transition between the text detection boxes is smooth; if the difference in coverage between a certain row in the preliminary row division results and the reference row on the horizontal axis is less than a set range threshold, and the distance between each text detection box in the row and the nearest text detection box in the reference row is less than a set distance threshold, then that row is added to the reference row group as a reference row.

[0029] After obtaining the reference row group, the initial line division is re-divided according to the reference row group, including: dividing all text detection boxes into n+1 regions from top to bottom according to the vertical axis based on the number n reference rows in the reference row group, and dividing the text detection boxes in each region according to the reference rows, including the following steps:

[0030] (a) Select the upper and lower reference rows for the current region from the reference row group. It should be noted that the uppermost and lowermost regions have only one reference row. It can be assumed that the other reference row is empty. The reference row with the smallest sum of distances to all text detection boxes in the current region is the main reference row (if it is an empty reference row, the distance between all text detection boxes and the reference row is set to infinity), and the other is the secondary reference row.

[0031] (b) Group the text detection boxes according to the length from the main reference line, group the text detection boxes with a distance difference within 0.5 times the line height, and sort the text detection box groups according to the distance difference;

[0032] (c) Determine whether a text detection box within 0.5 times the line height of the main reference line belongs to the main reference line. If so, assign it to the main reference line. The determination process can adopt a preliminary line division method, that is, as long as the text detection box and any text detection box in the main reference line are definitely not in the same line (e.g., the horizontal coordinate direction overlaps or the angle between the two boxes is greater than a certain angle), the two text detection boxes are temporarily considered to belong to the same line.

[0033] (d) For each group of text detection boxes, sort the text detection boxes in the group according to the horizontal axis, and then divide the text detection boxes in the group into lines according to the initial line division method;

[0034] (e) Determine whether a text detection box within 0.5 times the line height of the secondary reference line belongs to the secondary reference line. If so, assign it to the secondary reference line.

[0035] At this point, the process of re-split the lines is complete. To make the line splitting more perfect, the process of building reference line groups and re-splitting lines based on reference line groups can be repeated until the line splitting results no longer change.

[0036] S130: Determine the column type based on the re-rowing results, and group and split the columns according to the column type.

[0037] In this embodiment, the process of determining the column type based on the re-rowing result includes: analyzing each row from top to bottom based on the re-rowing result; if the difference between the number of text detection boxes in the current row and the previous row is less than a set number threshold, and the intersection of each text detection box and the nearest text detection box in the previous row on the horizontal axis exceeds a certain threshold, then it is determined that the current row and the previous row belong to the same column type.

[0038] After obtaining the column types, the text detection boxes are grouped and split according to the column types, including the following process: Taking the first row of each column type as a reference, traverse each row of the column type from top to bottom, and match all text detection boxes in the current row with all text boxes in the previous row. If a match is found, the matching text boxes are grouped into one column; otherwise, a new column is added. This yields the row and column results of the text detection boxes. The Hungarian algorithm is used to match all text detection boxes in the current row with all text boxes in the previous row, and the cost function is set as the distance between the two text boxes.

[0039] The text detection-based method for extracting and analyzing structured information from table images provided by this invention relies solely on the coordinates of the text detection box, specifically the relative horizontal and vertical coordinates. It is independent of the text recognition results or the presence of auxiliary lines, and supports various text detection box formats, such as rectangles, rotated rectangles, polygons, or polygons enclosed by Bézier curves. Furthermore, this invention exhibits good adaptability to curved and wrinkled images, making it suitable for intelligent structured information extraction and analysis of most table images. This invention requires no manual annotation and involves minimal computation, making it suitable for most computing devices and requiring no additional hardware resources.

[0040] Based on the same inventive concept, the embodiment also provides a table image structured information extraction and analysis device 200 based on text detection, including a preliminary row division module 210, a re-row division module 220, and a column division module 230. The preliminary row division module 210 is used to perform text detection on the table image to obtain the position information of each text detection box, and perform preliminary row division based on the position information of the text detection boxes to obtain a preliminary row division result. The re-row division module 220 is used to select multiple reference rows from the preliminary row division result and construct a reference row group, and perform re-row division on the preliminary row division according to the reference row group to obtain a re-row division result. The column division module 230 is used to determine the column type according to the re-row division result and group and columnize according to the column type.

[0041] It should be noted that the table image structured information extraction and analysis device based on text detection provided in the above embodiments should be illustrated using the above-described functional module divisions when performing table image structured information extraction and analysis. The functions can be assigned to different functional modules as needed, i.e., the internal structure of the terminal or server can be divided into different functional modules to complete all or part of the functions described above. Furthermore, the table image structured information extraction and analysis device based on text detection provided in the above embodiments and the table image structured information extraction and analysis method based on text detection belong to the same concept. For details of its implementation process, please refer to the table image structured information extraction and analysis method based on text detection, which will not be repeated here.

[0042] In practical applications, computer memory can be volatile memory located at the local end, such as RAM; it can also be non-volatile memory, such as ROM, FLASH, floppy disks, hard disks, etc.; or it can be a remote storage cloud. The computer processor can be a central processing unit (CPU), microprocessor (MPU), digital signal processor (DSP), or field-programmable gate array (FPGA). These processors can be used to implement the steps of extracting and analyzing structured information from table images based on text detection.

[0043] The specific embodiments described above illustrate the technical solution and beneficial effects of the present invention in detail. It should be understood that the above description is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for extracting and analyzing information of structured table image based on character detection, characterized in that, Includes the following steps: Text detection is performed on the table image to obtain the position information of each text detection box. Based on the position information of the text detection boxes, preliminary line division is performed to obtain the preliminary line division result. Select multiple reference rows from the initial row division results and construct a reference row group, including: selecting reference rows from the initial row division results and adding them to the reference row group based on the condition that the number of text detection boxes in the initial row division list reaches a threshold and the transition of the angle between the text detection boxes is smooth; if the difference in the coverage area between a row in the initial row division results and the reference row on the horizontal axis is less than a set range threshold, and the distance between each text detection box in the row and the nearest text detection box in the reference row is less than a set distance threshold, then that row is added to the reference row group as a reference row. The initial line breaks are re-broken based on the reference line group to obtain the re-broken line results; Determine the column type based on the results of the re-row splitting, and then group and split the columns according to the column type. 2.The method of claim 1, wherein, The preliminary line division based on the position information of the text detection box includes: Sort all text detection boxes into a list based on their horizontal coordinates. Iterate through the sorted list and compare the current text detection box with the existing preliminary line break list. If a match is found, store the current text detection box in the existing preliminary line break list. If no match is found, generate a new preliminary line break list for the current text detection box and store it. 3.The method of claim 2, wherein, The matching and judgment of the current text detection box being traversed with the existing preliminary line break list includes: Based on the relative positions of the two text detection boxes, determine whether the current text detection box and the last text detection box in the existing preliminary line break list belong to the same line. If so, use the existing preliminary line break list where the last text detection box is located as the current text detection box as the candidate line break list. If there is only one candidate line break list corresponding to the current text detection box, then the candidate line break list is considered to be the existing preliminary line break list matched by the current text detection box. When there are multiple candidate line breaks corresponding to the current text detection box, select the candidate line break list with the closest relative position as the existing preliminary line break list for matching the current text detection box. 4.The method of claim 3, wherein, Determining the relative position of two text detection boxes includes: If the horizontal coordinates of two text detection boxes overlap, or if the angle between the two text detection boxes is greater than the threshold, then the two text detection boxes are considered not to belong to the same row; otherwise, they are considered to belong to the same row. The relative position of the two text detection boxes is considered to be the smallest when the angle between them is the smallest, or when the angle change is the smallest.

5. The method for extracting and analyzing structured information from table images based on text detection according to claim 1, characterized in that, The step of re-framing the initial line breaks based on the reference line group includes: Based on the number of reference rows n in the reference row group, all text detection boxes are divided into n+1 regions from top to bottom according to the vertical axis. The text detection boxes in each region are then divided into rows according to the reference rows, including the following steps: Select two reference rows from the reference row group for the current region, with the reference row having the smallest sum of distances to all text detection boxes in the current region as the primary reference row and the other as the secondary reference row; The text detection boxes are grouped according to their distance from the main reference line. The text detection boxes with a distance difference of less than 0.5 times the line height are grouped together, and the text detection box groups are sorted according to the distance difference. Determine whether a text detection box within 0.5 times the line height of the main reference line belongs to the main reference line; if so, assign it to the main reference line. For each group of text detection boxes, sort the text detection boxes in the group according to the horizontal axis, and then divide the text detection boxes in the group into lines according to the initial line division method; Determine whether a text detection box within 0.5 times the line height of the secondary reference line belongs to the secondary reference line. If so, assign it to the secondary reference line.

6. The method for extracting and analyzing structured information from table images based on text detection according to claim 1, characterized in that, The determination of column type based on the result of re-row splitting includes: Based on the re-rowing results, each row is analyzed from top to bottom. If the difference between the number of text detection boxes in the current row and the previous row is less than a set threshold, and the intersection of each text detection box with the nearest text detection box in the previous row on the horizontal axis exceeds a certain threshold, then the current row and the previous row are determined to belong to the same column type.

7. The method for extracting and analyzing structured information from table images based on text detection according to claim 1, characterized in that, The grouping and splitting according to column type includes: Based on the first row of each column type, traverse each row of the column type from top to bottom, and match all text detection boxes in the current row with all text boxes in the previous row. If a match is found, group the matching text boxes into one column; otherwise, add a new column.

8. The method for extracting and analyzing structured information from table images based on text detection according to claim 7, characterized in that, The Hungarian algorithm is used to match all text detection boxes in the current row with all text boxes in the previous row, and the cost function is set as the distance between the two text boxes.

9. A device for extracting and analyzing structured information from table images based on text detection, characterized in that, Includes initial line splitting module, re-line splitting module, and column splitting module. The preliminary line division module is used to perform text detection on the table image to obtain the position information of each text detection box, and perform preliminary line division based on the position information of the text detection boxes to obtain the preliminary line division result; The re-line splitting module is used to select multiple reference rows from the preliminary line splitting results and construct a reference row group, including: selecting reference rows from the preliminary line splitting results and adding them to the reference row group based on the condition that the number of text detection boxes in the preliminary line splitting list reaches a threshold and the angle transition between the text detection boxes is smooth; if the difference in coverage between a certain row in the preliminary line splitting results and the reference row on the horizontal axis is less than a set range threshold, and the distance between each text detection box in the row and the nearest text detection box in the reference row is less than a set distance threshold, then that row is added to the reference row group as a reference row; It is also used to re-split the initial line breaks based on the reference line group to obtain the re-split results; The column splitting module is used to determine the column type based on the re-row splitting result, and to group and split the columns according to the column type.

Images