A method, system and storage medium for table recognition based on morphological detection

Abstract

The application discloses a table recognition method and system based on morphological detection and a storage medium, and belongs to the field of image processing, and solves the problems of the detection accuracy and anti-interference of the prior art in the detection of complex structure tables. The method comprises the following steps: step S1, locating a table region in a document image through a target detection model; step S2, detecting table straight lines through a Hough transform, and correcting the table direction according to the straight line inclination angle; step S3, after detecting the horizontal table lines and the vertical table lines constituting the table through morphological operation, the horizontal table lines and the vertical table lines are merged; step S4, analyzing the layout of the detected table lines, and classifying the table lines belonging to one table; step S5, analyzing the table structure, and calculating effective cell information; and step S6, recognizing the words in the cells, and saving the words into an editable file in combination with the cell positions.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image processing, in particular to a table recognition method and system based on morphological detection and a storage medium. BACKGROUND

[0002] As a structured data arrangement, the table has complex and non-fixed layout and contains important structured information. Recognizing the table structure and text content in the document image and converting them into editable format is conducive to document information retrieval and query in later stage, and further, is conducive to improving the processing efficiency of the document. The text content recognition is generally processed by mature OCR technology, while the table structure analysis is a difficulty in document image recognition.

[0003] The process of table structure recognition is to first recognize and extract the horizontal and vertical line segments constituting the table, and then analyze the structure of the table according to the extracted line segments. The existing table structure recognition methods include: a detection method based on Hough transform, a detection method based on projection, a method based on connected chain and a method based on run detection. Among them, the Hough transform method detects the horizontal and vertical lines in the image respectively, and then analyzes the structure of the table according to the characteristic that the horizontal and vertical lines in the table intersect perpendicularly. The disadvantage is that a large number of line segments unrelated to the table, such as line segments in text, formula and image, will be detected together, which will greatly interfere with the analysis of the table structure. The projection method analyzes the table structure by horizontal and vertical projection according to the characteristic that there is a large spacing between the rows and columns of the table. This method cannot effectively analyze the structure of the table with non-standard layout and merged cells, and is also affected to a certain extent when the table is inclined. In addition, the method based on connected chain detects the corresponding horizontal and vertical line segments through horizontal and vertical connected chains. The disadvantage is that the large amount of calculation results in slow detection speed. The principle of run detection is similar to that of connected chain. Run is defined as a group of continuous pixels in horizontal or vertical direction. Similarly, the corresponding table line segments are detected through horizontal and vertical run detection. The disadvantage is that it is easily affected by the broken table lines.

[0004] Therefore, the existing technology has the following defects: in the processing of complex structure table, the detection accuracy and anti-interference are insufficient. SUMMARY

[0005] The present application solves the problem of insufficient detection accuracy and anti-interference of the existing technology in the detection of complex structure table.

[0006] The table recognition method based on morphological detection provided by the present application comprises the following steps:

[0007] Step S1, locating the table region in the document image through a target detection model;

[0008] Step S2, detecting the table straight line by Hough transform, and correcting the table direction according to the straight line inclination angle;

[0009] Step S3, detecting the horizontal table line and the vertical table line constituting the table by morphological operation, and merging the horizontal table line and the vertical table line;

[0010] Step S4, analyzing the layout of the detected table line, and classifying the table line belonging to one table;

[0011] Step S5, analyzing the table structure, and calculating the effective cell information;

[0012] Step S6, recognizing the text in the cell, and saving as an editable file combined with the cell position.

[0013] Further, in the embodiment of the application, in the step S1, the target detection model is YOLOv5, which is used to detect the position of the table in the image to obtain the position information of the table;

[0014] The position information is represented as (x0, y0, w, h), wherein x0 and y0 are the position coordinates of the upper left corner of the table region in the image, and w and h are the width and height of the table region in the image, respectively.

[0015] Further, in the embodiment of the application, in the step S3, the horizontal table line and the vertical table line constituting the table are detected by morphological operation, including the following steps:

[0016] Step S301, first converting the document image into a gray-scale image, and then performing binaryzation processing on the gray-scale image, wherein the pixel value of the document background part in the processing result is 0, and the pixel value of the non-background part such as text and table is 255;

[0017] Step S302, performing horizontal and vertical morphological opening operation processing on the binary image, respectively, which is composed of morphological erosion and morphological dilation operations, wherein the size of the structure element of the erosion and dilation operations is (w / s, 1) when performing horizontal processing, and the size of the structure element is (1, h / s) when performing vertical processing, wherein h represents the height of the image, w represents the width of the image, and s represents the scaling factor;

[0018] Step S303, for the horizontal and vertical morphological detection results, the starting position of the line segment with a continuous pixel value of 255 in each row and each column is detected, that is, the table line, to obtain the set of horizontal table lines and vertical table lines.

[0019] Further, in the embodiment of the application, in the step S3, the horizontal table line and the vertical table line are merged, including the following steps:

[0020] Step S304, filtering the line segments with length below a set threshold, and removing the detected character composed line segments;

[0021] Step S305, traversing the line segments after the preliminary filtering, and judging whether two line segments satisfy a merging condition.

[0022] Further, in the embodiment of the present application, in the step S305, the merging condition is that:

[0023] the distance between the two line segments is less than a set distance threshold, and the included angle formed by the two line segments is less than a set angle threshold.

[0024] Further, in the embodiment of the present application, in the step S4, the analysis of the layout of the detected table lines comprises the following steps:

[0025] Step S401, sorting the horizontal table lines according to their vertical coordinates in ascending order, which corresponds to the order from top to bottom in the image, and sorting the vertical table lines according to their horizontal coordinates in ascending order, which corresponds to the order from left to right in the image;

[0026] Step S402, starting from the first horizontal table line segment, traversing all the horizontal table lines downwards, and judging whether they belong to the same table according to horizontal line same group judgment condition, if there are multiple tables in the image, multiple horizontal line groups can be divided, and for each horizontal line group, all the vertical table lines are traversed according to the vertical line same group judgment condition, and the vertical table lines which form the table together with the horizontal line group are taken out;

[0027] Step S403, returning the vertical coordinate set of the horizontal line segments and the horizontal coordinate set of the vertical line segments which constitute each table.

[0028] Further, in the embodiment of the present application, in the step S402, the horizontal line same group judgment condition is whether there is a vertical line which intersects with two horizontal lines at the same time.

[0029] The vertical line same group judgment condition is that the vertical line intersects with two or more horizontal lines in the horizontal line group.

[0030] Further, in the embodiment of the present application, in the step S5, the analysis of the table structure comprises the following steps:

[0031] Step S501, calculating the information of each minimum cell in the table according to the vertical coordinate set of the horizontal line segments and the horizontal coordinate set of the vertical line segments which constitute each table.

[0032] Step S502, taking each minimum cell as a candidate, checking whether the four edges constituting the candidate cell exist in the table line in step S4, if they exist, it is a valid cell, if they do not exist, it is a missing table line, for the cell corresponding to the table combined by multiple cells, further combine the table lines around this cell as new candidate cells, continue to check their four edges, if multiple candidate cells sharing the same top-left corner point are found, only the smallest cell is retained as a valid cell.

[0033] The table recognition system based on morphological detection comprises:

[0034] A region module, which is used to locate the table region in the document image through a target detection model;

[0035] A direction module, which is used to detect the table straight line through Hough transformation and correct the table direction according to the straight line inclination angle;

[0036] A merging module, which is used to merge the horizontal table lines and the vertical table lines after detecting the horizontal table lines and the vertical table lines through morphological operation;

[0037] A table line module, which is used to analyze the layout of the detected table lines and classify the table lines belonging to one table;

[0038] An information module, which is used to analyze the table structure and calculate the valid cell information;

[0039] An editing module, which is used to recognize the text in the cell and save it as an editable file in combination with the cell position.

[0040] The computer readable storage medium comprises a computer program stored thereon, and the computer program is executed by a processor to realize the steps of any one of the above methods.

[0041] The present application solves the problems of the prior art in the detection accuracy and anti-interference of complex structure tables.

[0042] The table recognition method based on morphological detection can accurately locate the table region in the document image through a target detection model, and on this basis, the table line can be detected through a morphological detection method, which can effectively exclude the interference of non-table line elements (such as header horizontal lines, formulas and illustrations) in the document image; further table line analysis can effectively identify the table structure, and can also effectively identify the complex structure table with table merging. Thus, the detection accuracy of the prior art for complex structure tables is improved, and the anti-interference of table detection is improved. BRIEF DESCRIPTION OF DRAWINGS

[0043] The above and / or additional aspects and advantages of the present application will become apparent and more readily appreciated from the following description, taken in conjunction with the following drawings of which:

[0044] Figure 1 is a flowchart of a table recognition method based on morphological detection according to the present application.

[0045] Figure 2 is a table positioning result map according to the specific embodiment.

[0046] Figure 3 is a table recognition result map according to the specific embodiment. DETAILED DESCRIPTION

[0047] Various embodiments of the present application will be described below in detail with reference to the accompanying drawings. The embodiments described by reference to the accompanying drawings are exemplary and are intended to explain the present application, and should not be understood as limiting the present application.

[0048] A table recognition method based on morphological detection according to the present embodiment, with reference to Figure 1 The present embodiment can be better understood, including the following steps:

[0049] Step S1, locating the table region in the document image by a target detection model;

[0050] Step S2, detecting table straight lines by Hough transform, and correcting the table direction according to the straight line inclination angle;

[0051] Step S3, after detecting the horizontal table lines and the vertical table lines that constitute the table by morphological operation, merging the horizontal table lines and the vertical table lines;

[0052] Step S4, analyzing the layout of the detected table lines, and classifying the table lines belonging to one table;

[0053] Step S5, analyzing the table structure, and calculating the effective cell information;

[0054] Step S6, recognizing the text in the cell, and saving it as an editable file in combination with the cell position.

[0055] In the present embodiment, in the step S1, the target detection model is YOLO v5, which is used to detect the position of the table in the image, and obtain the position information of the table;

[0056] The position information is represented as (x0, y0, w, h), wherein x0 and y0 are the position coordinates of the top left corner of the table region in the image, and w and h are the width and height of the table region in the image, respectively.

[0057] In the step S3, the detecting the horizontal table lines and the vertical table lines by morphological operation includes the following steps:

[0058] In the step S301, the document image is firstly converted into a gray image, and then the gray image is binarized, in which the pixel value of the document background is 0, and the pixel value of the non-background such as the text and the table is 255;

[0059] In the step S302, the horizontal and vertical morphological opening operation is respectively performed on the binarized image, which is composed of morphological erosion and morphological dilation operation, wherein the size of the structure element of the erosion and dilation operation is (w / s, 1) in the horizontal processing, and the size of the structure element is (1, h / s) in the vertical processing, wherein h represents the height of the image, w represents the width of the image, and s represents the scaling factor;

[0060] In the step S303, the starting position of the line segment with the continuous pixel value of 255 in each row and each column is detected as the table line, so as to obtain the set of the horizontal table lines and the vertical table lines.

[0061] In the step S3, the merging the horizontal table lines and the vertical table lines includes the following steps:

[0062] In the step S304, the line segment with the length lower than the set threshold value is filtered, and the detected text line segment is removed.

[0063] In the step S305, the two line segments are judged whether to satisfy the merging condition by traversing the line segments after the preliminary filtering.

[0064] In the step S305, the merging condition is that:

[0065] The distance between the two line segments is less than the set distance threshold value, and the included angle formed by the two line segments is less than the set angle threshold value.

[0066] In the step S4, the analyzing the layout of the detected table lines includes the following steps:

[0067] In the step S401, the horizontal table lines are sorted in ascending order according to the vertical coordinates, and the vertical table lines are sorted in ascending order according to the horizontal coordinates.

[0068] Step S402, starting from the first horizontal table line segment, traversing all horizontal table lines, judging whether they belong to the same table according to the horizontal line same group judgment condition. If there are multiple tables in the image, multiple horizontal line groups can be divided. For each horizontal line group, traverse all vertical table line segments according to the vertical line same group judgment condition, and take out the vertical table line that constitutes the table together with the horizontal line group.

[0069] Step S403, return the vertical coordinate set of the horizontal line segment and the horizontal coordinate set of the vertical line segment that constitute each table.

[0070] In this embodiment, in step S402, the horizontal line same group judgment condition is whether there is a vertical line that intersects with two horizontal lines at the same time.

[0071] The vertical line same group judgment condition is that the vertical line intersects with two or more horizontal lines in the horizontal line group.

[0072] In this embodiment, in step S5, the analysis of the table structure includes the following steps:

[0073] Step S501, according to the vertical coordinate set of the horizontal line segment and the horizontal coordinate set of the vertical line segment that constitute each table, calculate the information of each minimum cell in the table;

[0074] Step S502, take each minimum cell as a candidate, check whether the four edges constituting the candidate cell exist in the table line in step S4. If they exist, it is a valid cell. If they do not exist, it is a missing table line. For the cell combined by multiple cells in the corresponding table, further combine the table lines around the cell as new candidate cells, and continue to check their four edges. If multiple candidate cells sharing the same top-left corner point are found, only the smallest cell is retained as a valid cell.

[0075] A table recognition system based on morphological detection, the system comprises:

[0076] A region module locates the table region in the document image through a target detection model;

[0077] A direction module detects table straight lines through Hough transform and corrects the table direction according to the inclination angle of the straight lines;

[0078] A merging module merges the horizontal table lines and the vertical table lines after detecting the horizontal table lines and the vertical table lines through morphological operation;

[0079] A table line module analyzes the layout of the detected table lines and classifies the table lines belonging to one table;

[0080] An information module, analyzing a table structure, calculating effective cell information;

[0081] An editing module, identifying text in a cell, and saving as an editable file in combination with a cell position.

[0082] A computer readable storage medium according to the embodiment, wherein the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of the method according to any one of the above embodiments.

[0083] A computer device according to the embodiment, comprising a memory and a processor, wherein the memory stores a computer program, and the processor executes the steps of the method according to any one of the above embodiments when the computer program stored in the memory is executed.

[0084] The embodiment provides an actual implementation of the table recognition method based on morphological detection according to the application.

[0085] Step S1, locating a table region in a document image by a target detection model; specifically, an image used for description is taken from a table recognition public dataset Marmot Dataset, and the image size is selected as: width 468, height 720, and the table positioning result is as shown in Figure 2 .

[0086] Step S2, detecting table straight lines by a Hough transform, and correcting a table direction according to a straight line inclination angle;

[0087] Step S3, after detecting horizontal table lines and vertical table lines constituting a table by morphological operation, merging the horizontal table lines and the vertical table lines;

[0088] Step S4, analyzing layout of the detected table lines, and classifying table lines belonging to one table;

[0089] Step S5, analyzing a table structure, and calculating effective cell information;

[0090] Step S6, identifying text in a cell, and saving as an editable file in combination with a cell position. The recognition result is as shown in Figure 3 .

[0091] In the step S1, the target detection model is YOLO v5, which is used to detect the position of the table in the image to obtain the position information of the table. The position information is represented as (x0, y0, w, h), wherein x0 and y0 are the position coordinates of the top left corner of the table region in the image, and w and h are the width and height of the table region in the image, respectively. Specifically, the obtained position information is (13, 384, 479, 189).

[0092] The step S3 comprises the following steps of:

[0093] In step S301, the document image is firstly converted into a gray image, and then the gray image is binarized, wherein the pixel value of the document background part is 0, and the pixel value of the non-background part such as the text and the table is 255;

[0094] In step S302, the binarized image is processed by horizontal and vertical morphological opening operations, respectively, and the morphological opening operation is composed of morphological erosion and morphological dilation operations, wherein the size of the structure element of the erosion and dilation operations is (w / s, 1) in the horizontal processing, and the size of the structure element is (1, h / s) in the vertical processing, wherein h represents the height of the image, w represents the width of the image, and s represents a scaling factor;

[0095] In step S303, the starting positions of the line segments with the continuous pixel value of 255 in each row and each column are detected as the table lines, and the horizontal table lines and the vertical table lines are obtained.

[0096] The step S3 comprises the following steps of:

[0097] In step S304, the line segments with the length lower than the set threshold value are filtered, and the detected text line segments are removed.

[0098] In step S305, the line segments after the preliminary filtering are traversed, and it is judged whether two line segments satisfy the merging condition.

[0099] The merging condition in step S305 is that:

[0100] The distance between the two line segments is less than the set distance threshold value, and the included angle formed by the two line segments is less than the set angle threshold value.

[0101] The step S4 comprises the following steps of:

[0102] In step S401, the horizontal table lines are sorted in ascending order according to their longitudinal coordinates, and the vertical table lines are sorted in ascending order according to their horizontal coordinates.

[0103] Step S402, starting from the first horizontal table line segment, traversing all horizontal table lines downwards, judging whether they belong to the same table according to the horizontal line same group judging condition, if there are multiple tables in the image, thus multiple horizontal line groups can be divided, for each horizontal line group, traversing all vertical table line segments according to the vertical line same group judging condition, taking out the vertical table line which constitutes the table together with the horizontal line group;

[0104] Step S403, returning the vertical coordinate set of the horizontal line segment and the horizontal coordinate set of the vertical line segment which constitute each table.

[0105] In step S402, the horizontal line same group judging condition is: whether there is a vertical line which intersects with two horizontal lines at the same time;

[0106] The vertical line same group judging condition is: the vertical line intersects with two or more horizontal lines in the horizontal line group.

[0107] In step S5, the analysis of the table structure includes the following steps:

[0108] Step S501, according to the vertical coordinate set of the horizontal line segment and the horizontal coordinate set of the vertical line segment which constitute each table, calculating the information of each minimum cell in the table;

[0109] Step S502, taking each minimum cell as a candidate, checking whether the four edges of the candidate cell exist in the table line in step S4, if they exist, it is a valid cell, if they do not exist, it is a missing table line, for the cell which is combined by multiple cells in the table, further combining the table line around the cell as a new candidate cell, continuing to check its four edges, if multiple candidate cells which share the same top-left corner point are found, only the smallest one is kept as a valid cell.

[0110] The principles and implementation manners of the present application are described by using specific examples in this paper, the above examples are only used to help understand the method of the present application and its core idea; meanwhile, for the general technical personnel in this field, according to the idea of the present application, the specific implementation manner and application range can be changed. In conclusion, the content of the present specification should not be understood as the limitation of the present application.

Claims

1. A table recognition method based on morphological detection, characterized in that, Includes the following steps: Step S1: Locate the table region in the document image using the object detection model; Step S2: Detect straight lines in the table using Hough transform, and correct the table orientation based on the tilt angle of the straight lines; Step S3: After detecting the horizontal and vertical table lines that make up the table through morphological operations, merge the horizontal and vertical table lines. The merging conditions are as follows: The distance between the two line segments is less than the set distance threshold, and the angle formed by the two line segments is less than the set angle threshold. Step S4: Analyze the detected table line layout and classify the table lines belonging to one table; In step S4, the analysis and detection of table line layout includes the following steps: Step S401: Sort the horizontal table lines in ascending order of their vertical coordinates, corresponding to a top-to-bottom order in the image; sort the vertical table line segments in ascending order of their horizontal coordinates, corresponding to a left-to-right order in the image. Step S402: Starting from the first horizontal table line segment, traverse all horizontal table lines downwards. Determine whether they belong to the same table based on the group discrimination condition of horizontal lines. If there are multiple tables in the image, multiple horizontal line combinations can be divided. For each horizontal line combination, traverse all vertical table line segments based on the group discrimination condition of vertical lines, and extract the vertical table line that forms the table together with the horizontal line group. In step S402, the condition for determining whether horizontal lines are in the same group is: whether there is a vertical line that intersects two horizontal lines simultaneously. The condition for determining whether a vertical line belongs to the same group is that the vertical line intersects with two or more horizontal lines in the same group. Step S403: Return the set of ordinates of the horizontal line segments and the set of abscissas of the vertical line segments that make up each table; Step S5: Analyze the table structure and calculate the valid cell information; Step S6: Recognize the text within the cell and, based on the cell position, save it as an editable file.

2. The table recognition method based on morphological detection according to claim 1, characterized in that, In step S1, the target detection model is YOLO v5, which is used to detect the position of the table in the image and obtain the position information of the table. The location information is represented as (x0, y0, w, h), where x0 and y0 are the coordinates of the top left corner of the table area in the image, and w and h are the width and height of the table area in the image, respectively.

3. The table recognition method based on morphological detection according to claim 1, characterized in that, In step S3, the detection of the horizontal and vertical table lines constituting the table through morphological operations includes the following steps: Step S301: First, convert the document image to a grayscale image, and then perform binarization on the grayscale image. In the processing result, the pixel value of the document background part is 0, and the pixel value of the text, table and other non-background parts is 255. Step S302: Perform morphological opening operations on the binary image in both the horizontal and vertical directions. The morphological opening operation consists of morphological erosion and morphological dilation operations. In the horizontal processing, the structuring element size of the erosion and dilation operation is (w / s, 1), and in the vertical processing, the structuring element size is (1, h / s), where h represents the height of the image, w represents the width of the image, and s represents the scaling factor. Step S303: For the morphological detection results of the horizontal and vertical directions, detect the starting position of the line segment with a continuous pixel value of 255 in each row and each column, which is the table line, and obtain the set of horizontal table lines and vertical table lines.

4. The table recognition method based on morphological detection according to claim 1, characterized in that, In step S3, merging the horizontal and vertical table lines includes the following steps: Step S304: Filter line segments with a length lower than a set threshold and remove detected line segments composed of text. Step S305: Traverse the pre-filtered line segments and determine whether two line segments meet the merging condition.

5. The table recognition method based on morphological detection according to claim 1, characterized in that, In step S5, the analysis table structure includes the following steps: Step S501: Calculate the information of each smallest cell in the table based on the set of horizontal and vertical coordinates that make up each table and the set of horizontal coordinates of the vertical line segments. Step S502: Take each smallest cell as a candidate and check whether the four sides of the candidate cell exist in the table lines in step S4. If they exist, the cell is valid; if they do not exist, the table lines are missing. For cells in the table that are merged from multiple cells, merge the table lines around the cell to make a new candidate cell. Continue to check its four sides. If multiple candidate cells that share the same top-left corner are found, only the smallest cell is kept as a valid cell.

6. A table recognition system based on morphological detection, characterized in that, The system includes: The region module uses an object detection model to locate table regions in a document image; The orientation module detects straight lines in the table using Hough transform and corrects the table orientation based on the tilt angle of the lines. The merging module detects the horizontal and vertical table lines that make up the table through morphological operations and then merges the horizontal and vertical table lines. The table line module analyzes and detects the layout of table lines, categorizing them into table lines belonging to a single table. The information module analyzes the table structure and calculates the information of valid cells. The editing module recognizes the text within cells and, based on the cell position, saves it as an editable file.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

Images