Wired table identification method based on morphology

Abstract

The invention provides a wired table recognition method based on morphology, and the method comprises the following steps: S1, carrying out the graying processing of all read-in to-be-detected three-channel images, obtaining a gray image, carrying out the binarization processing of the obtained gray image, and obtaining a binary image of an original image; and S2, carrying out corrosion and expansion operation on the binary image by using a detection core, and forming an Excel format file by identifying a text table format and the like in the image so as to solve the problem of a special-shaped table. The problem that in the prior art, table recognition is always a research difficulty in the field of document recognition is solved, and the method is high in running speed, high in stability and good in interpretability. And the method is innovative, and can be used for well processing conditions such as special-shaped tables, cross-page disconnection, text cross-line, text wireless frame package and the like. The method can be popularized, and has good compatibility for identification of different types of wired tables. The method can be deployed in a server, a personal pc (personal computer) and embedded equipment, and is high in practical applicability.

Description

Technical Field

[0001] The present invention relates to the technical field of table recognition, and more particularly to a wired table recognition method based on morphology. Background Art

[0002] There is a huge market demand for cross-media conversion from images to structured data. Taking general OCR as an example, there is a huge demand for identifying text, images, and tables in document images (especially scanned ones). In the case of pure text, you only need to detect the box and complete the recognition. When it comes to tables, you also need to calculate the logical relationship between text and text before you can restore it to Excel, which is a huge challenge. Table recognition is mainly divided into three parts: a. Table detection: Wired tables need to detect the coordinates of the four corner points of the table. b. OCR recognition: Detect the circumscribed rectangular box of the text in the table and recognize the text content. c. Table restoration: First, bind the table detection result and the OCR recognition result through the coordinate space relationship, and then calculate the relative position relationship according to the coordinates of the table to form a logical structure and restore Excel.

[0003] There are many deep learning models for end-to-end prediction of logical structures in the existing technology. However, under the premise of limited training data, the performance is often unsatisfactory when switching to a type of table. The data annotation of logical structures is particularly complex and inefficient, requiring a lot of manpower and time costs. The existing technical solutions have the following shortcomings:

[0004] a. Scanned or photographed versions of the table may be distorted and difficult to restore without correction. Figure 2 As shown, the red frame is the correction target.

[0005] b. When a cell edge is missing or has a long break, it is difficult to detect it morphologically, resulting in the loss of the cell. Figure 3 As shown, the first row will be missed.

[0006] c. In order to improve recognition efficiency, an OCR will be performed on all text boxes in the large image (it will be very time-consuming to perform OCR detection inside each cell), but the OCR detection box cannot fit the text 100%, and when the text belonging to different cells is close, it will usually be detected as a text box, and the binding of OCR and table results will be confused. Figure 4 With the instruction manual Figure 5 In the two cases shown, the red OCR detection box spans multiple cells.

[0007] d. Morphological methods mainly detect straight lines in tables, but Chinese characters have many horizontal and vertical strokes, which can easily lead to false detections.

[0008] As the instruction manual Figure 6 The red vertical line is shown.

[0009] e. Irregular-shaped tables usually have many shorter horizontal or vertical lines that are not collinear with other line segments. The main difficulty of this type of table is calculating the logical relationship of each cell.

[0010] As the instruction manual Figure 7 As shown, the red line is the extension line of the original table segment. It can be seen from the figure that many segments have such problems.

[0011] f. There are also wireless tables inside wired tables. If all text boxes in the cells are directly connected into a string without processing, the recovery effect will be poor. Figure 8 The first row is shown. Summary of the invention

[0012] The technical problem to be solved by the present invention is to provide a morphologically based wired table recognition method that is interpretable, does not require labeled data, and is efficient and high-precision.

[0013] The present invention provides a wired table recognition method based on morphology, the method comprising the following steps:

[0014] S1, grayscale processing is performed on all the three-channel images to be detected to obtain grayscale images, and the obtained grayscale images are binarized to obtain binary images of the original images;

[0015] S2, performing erosion and expansion operations on the binary image through a detection core, and detecting the binary image after the erosion and expansion operations to obtain a table line binary image;

[0016] S3, performing image correction on the table line binary image and the three-channel image through a perspective matrix M;

[0017] S4, combining the table line binary images after image correction into a grid binary image, negating the grid binary image, and performing erosion and expansion operations on the negated grid binary image by detection and verification, using a contour search method to obtain closed cells on the grid binary image after the erosion and expansion operations, filtering out cells with too large or too small areas and cells included by other cells, and calculating the rows and columns of the list where the cells are located;

[0018] S5, using the OCR detection model to detect the text boxes in all the corrected three-channel images, performing IOU matching calculations on the text boxes and the cells, and determining whether the text boxes and the cells are successfully matched. If successful, the text boxes are cropped to obtain new text boxes, and the new text boxes are stored in a list. The OCR recognition model is used to perform text recognition on the text boxes in the list to obtain text results, and the text results are combined into strings from top to bottom and from left to right, and the rows and columns of the list are located according to the obtained text results.

[0019] S6, recovering the table line binary image and character string to obtain an excel file.

[0020] In the morphology-based wired table recognition method of the present invention, step S1 comprises the following steps:

[0021] S11, setting a pixel grayscale threshold T1, and using the threshold T1 to convert a 0-255 grayscale image into a binary image.

[0022] In the morphology-based wired table recognition method of the present invention, step S2 comprises the following steps:

[0023] S21, performing corrosion and expansion operations on the binary image by detecting the core to obtain a vertical line binary image;

[0024] S22, performing corrosion and expansion operations on the binary image by detecting and checking the binary image to obtain a horizontal line binary image.

[0025] In the morphology-based wired table recognition method of the present invention, step S3 comprises the following steps:

[0026] S31, set the target coordinates to (X0, Y0), (X1, Y1), (X2, Y2), (X3, Y3), and obtain the two horizontal lines with the minimum and maximum y coordinates of the horizontal line binary image and the width not less than 0.5*img_w and the two vertical lines with the minimum and maximum x coordinates of the vertical line binary image and the width not less than 0.5*img_h, and calculate the intersection of the two horizontal lines and the two vertical lines to obtain the intersection coordinates (x0, y0), (x1, y1), (x2, y2), (x3, y3), calculate the intersection coordinates (x0, y0), (x1, y1), (x2, y2), (x3, y3) and the target coordinates (X0, Y0), (X1, Y1), (X2, Y2), (X3, Y3) through perspective transformation to obtain the perspective matrix M, and use the perspective matrix M to perform image correction on the vertical line binary image, the horizontal line binary image and the three-channel image.

[0027] In the morphology-based wired table recognition method of the present invention, step S4 comprises the following steps:

[0028] S41. Multiply the vertical line binary image and the horizontal line binary image after image correction by 0.5 respectively, then add them together to form a grid binary image. Invert the grid binary image, and perform erosion and dilation operations on the inverted grid binary image through detection and verification. Use the contour search method on the grid binary image after erosion and dilation operations to obtain closed cells.

[0029] In the method for recognizing wired tables based on morphology according to the present invention; the step S4 further includes the following steps:

[0030] S42. Filter out cells with too large an area when the area of the cell satisfies w*h < img_w*img_h*alpha_1; where, w represents the width of the cell, h represents the height of the cell, img_w represents the width of the image, img_h represents the height of the image, and alpha_1 is a coefficient between 0 and 1;

[0031] S43. Filter out cells with too small an area when the area of the cell satisfies w*h > img_w*img_h*alpha_2, where, w represents the width of the cell, h represents the height of the cell, img_w represents the width of the image, img_h represents the height of the image, alpha_2 is a coefficient between 0 and 1, and 0 < alpha_2 < alpha_1;

[0032] S44. Obtain the minimum value CELL_MIN_H of the heights of all cells and the minimum value CELL_MIN_W of the widths. Sort all cells in ascending order according to the upper left corner y coordinate cell_min_y. Calculate the difference between the y coordinates cell_min_y of adjacent cells. If cell_min_y1 - cell_min_y0 is less than half of CELL_MIN_H, it is considered that the two cells belong to the same row, and min(cell_min_y1 - cell_min_y0) is stored as the y coordinate of this row in the row coordinate list row_y;

[0033] S45. Traverse the lower right corner y coordinate cell_max_y of all cells. If the difference between the lower right corner y coordinate cell_max_y and all rows in the row coordinate list row_y is greater than half of CELL_MIN_H, then update cell_max_y to the row coordinate list row_y;

[0034] S46, find the row with the most cells, store the x-coordinate cell_min_x of the upper left corner of each cell as the coordinate of each column in the column coordinate list col_x, traverse the x-coordinate cell_max_x of the lower right corner of all cells, if the difference between the x-coordinate cell_max_x and all columns in the column coordinate list col_x is greater than half of CELL_MIN_W, then update cell_max_x to the column coordinate list col_x.

[0035] In the morphology-based wired table recognition method of the present invention, step S5 comprises the following steps:

[0036] S51, match the text box with the cell to obtain the intersection-over-union ratio, wherein the IOU matching calculation method is inter(text, cell).area / text.area, wherein inter(text, cell).area is the area of ​​the intersection of the text box and the cell, and text.area is the area of ​​the text box; if the intersection-over-union ratio of the current text box and all cells is less than or equal to the set threshold T2, it is determined that the current text box is not in any cell, and the text box is cached in the no_match list; if the intersection-over-union ratio of the current text box and all cells is greater than or equal to (1-T2), it is determined that the text box belongs to this cell, and the text box will not be cut, and the text box is cached in the match list; if the intersection-over-union ratio of the current text box and all cells is greater than T2 and less than 1-T2, the intersection of the cell and the text box is cut out as a new text box, and the new text box is attributed to this cell. If the intersection-over-union ratio is less than 0.5, the new text box is stored in the match_cache list, otherwise it is stored in the match list.

[0037] In the morphology-based wired table recognition method of the present invention, the step S5 further includes the following steps:

[0038] S52, traverse all text boxes belonging to the same cell in the match, match_cache and no_match lists, use the OCR recognition model to perform text recognition on the text boxes to obtain text results, and combine the text results into a string from top to bottom and from left to right.

[0039] In the morphology-based wired table recognition method of the present invention, the step S5 further includes the following steps:

[0040] S53, traverse the text boxes in the no_match list, use the text box and the upper left corner coordinates of the cell (text_min_x, text_min_y) as reference points, and compare them with the column coordinate list col_x and the row coordinate list

[0041] row_y, if the coordinates of the upper left corner of a text box meet the following conditions at the same time: column coordinate list

[0042] If the i-th element in col_x is less than text_min_x and text_min_x is less than the i+1-th element in col_x; if the j-th element in the row coordinate list row_y is less than text_min_y and text_min_y is less than the j+1-th element in row_y, the coordinates of the upper left corner of the text box are updated to (col_x[i], row_y[j]);

[0043] S54, traverse the text boxes in the no_match list, take the coordinates of the lower right corner of the text box (text_max_x, text_max_y) as the reference point, and compare them with the column coordinate list col_x and the row coordinate list

[0044] row_y, if the coordinates of the lower right corner of a text box meet the following conditions at the same time: column coordinate list

[0045] If the i-th element in col_x is less than text_max_x and text_max_x is less than the i+1-th element in col_x; if the j-th element in the row coordinate list row_y is less than text_max_y and text_max_y is less than the j+1-th element in row_y, the coordinates of the upper left corner of the text box are updated to (col_x[j+1], row_y[i+1]);

[0046] S55, if the coordinates of the text box exceed the row coordinate list row_y or the column coordinate list col_x, expand outwards with the minimum row spacing and the minimum column spacing until the coordinates of the text box are covered;

[0047] S56, merge the text boxes with the same coordinates of the upper left corner and the lower right corner, and merge the text into a string from top to bottom and from left to right.

[0048] In the morphology-based wired table recognition method of the present invention, step S6 comprises the following steps:

[0049] S61, traverse the upper left corner coordinates (min_x, min_y) of the text box in the cell and no_match list, and compare the upper left corner coordinates (min_x, min_y) with the column coordinate list col_x and the row coordinate list row_y. If the following conditions are met at the same time: the absolute difference between min_y and the j-th element in the row coordinate list row_y is less than half of CELL_MIN_H; the absolute difference between min_x and the i-th element in the column coordinate list col_x is less than half of CELL_MIN_W, then add an entry to the logical structure DICT_L to indicate that the text box is located at the i-th row and j-th column.

[0050] S62, traverse the lower right corner coordinates (max_x, max_y) of the text box in the cell and no_match list, and compare the lower right corner coordinates (max_x, max_y) with the column coordinate list col_x and the row coordinate list row_y. If the following conditions are met at the same time: the absolute difference between max_y and the mth element in the row coordinate list row_y is less than half of CELL_MIN_H; the absolute difference between max_x and the nth element in the column coordinate list col_x is less than half of CELL_MIN_W; then update the position information of the text box in the logical structure DICT_L, and record the row and column where it is located and the number of cells (nj) that the text box needs to merge to the right and the number of cells (mi) that the text box needs to merge to the bottom;

[0051] S63, create a two-dimensional matrix M of size len(row_y)*len(col_x), the two-dimensional matrix M is used to record whether each cell has been filled, and initialize all elements to 0, indicating that no cell is filled, traverse the logical structure DICT_L, and update the matrix M according to the position information of each text box; if a cell is covered in the previous step, set the corresponding matrix element to 1;

[0052] S64, arrange the logical structure DICT_L in order from top to bottom by row i and from left to right by column j, generate an HTML file from the sorted logical structure DICT_L, use the read_html function of pandas to read the generated HTML file and convert it into an Excel file.

[0053] The morphology-based wired table recognition method of the present invention solves the problem of irregular tables by identifying the text table format in the picture, etc., to form an Excel format file. It solves the problem that the recognition of tables in the prior art has always been a research difficulty in the field of document recognition. The present application has fast running speed, strong stability and good interpretability. It is innovative and has good handling of irregular tables, cross-page breaks, text cross-lines, text wireless frame wrapping and other situations. It can be extended and has good compatibility for different types of wired table recognition. The present application can be deployed on servers, personal PCs, and embedded devices, and has strong practical applicability. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 It is a flow chart of an embodiment of a wired table recognition method based on morphology of the present invention;

[0055] Figure 2 It is a schematic diagram showing that the table image in the prior art of the morphology-based wired table recognition of the present invention may be distorted;

[0056] Figure 3 It is a schematic diagram of a cell edge missing or having a long break in the prior art of morphology-based wired table recognition of the present invention;

[0057] Figure 4 It is a schematic diagram of an OCR detection frame spanning multiple cells in the prior art of morphology-based wired table recognition of the present invention;

[0058] Figure 5 It is a schematic diagram of an OCR detection frame spanning multiple cells in the prior art of morphology-based wired table recognition of the present invention;

[0059] Figure 6 is a schematic diagram of a cell detection error in the prior art of morphology-based wired table recognition of the present invention;

[0060] Figure 7 It is a schematic diagram of the existence of multiple line segments in a unit in the prior art of the wired table recognition based on morphology of the present invention;

[0061] Figure 8 It is a schematic diagram of the situation in which a wireless table also exists inside a wired table in the prior art in the morphology-based wired table recognition of the present invention. DETAILED DESCRIPTION

[0062] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0063] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0064] like Figure 1 FIG. 1 is a flow chart of an embodiment of a wired table recognition method based on morphology of the present invention. A wired table recognition method based on morphology is provided, and the method comprises the following steps:

[0065] In step S1, grayscale processing is performed on all the three-channel images to be detected to obtain grayscale images, and the obtained grayscale images are binarized to obtain binary images of the original images;

[0066] In step S2, the binary image is corroded and expanded by a detection check, and the binary image after the corrosion and expansion operations is detected to obtain a table line binary image;

[0067] In step S3, the table line binary image and the three-channel image are corrected by using a perspective matrix M;

[0068] In step S4, the table line binary image after image correction is combined into a grid binary image, the grid binary image is inverted, and the inverted grid binary image is eroded and expanded by detection and verification, and the grid binary image after erosion and expansion is used to obtain closed cells using a contour search method, and cells with too large or too small areas and cells included by other cells are filtered out, and the rows and columns of the list where the cells are located are calculated;

[0069] In step S5, the text boxes in all the corrected three-channel images are detected by the OCR detection model, the IOU matching calculation is performed between the text box and the cell, and it is determined whether the text box and the cell are successfully matched. If successful, the text box is cropped to obtain a new text box, and the new text box is stored in a list. The OCR recognition model is used to perform text recognition on the text boxes in the list to obtain text results, and the text results are combined into a string from top to bottom and from left to right, and the rows and columns of the list are located according to the obtained text results;

[0070] In step S6, the table line binary image and character string are restored to obtain excel.

[0071] In one embodiment, the step S1 comprises the following steps:

[0072] In step S11, a pixel grayscale threshold T1 is set, and the grayscale image of 0-255 is converted into a binary image using the threshold T1.

[0073] In one embodiment, the step S2 comprises the following steps:

[0074] In step S21, the binary image is eroded and expanded by checking and checking to obtain a vertical line binary image;

[0075] In step S22, the binary image is eroded and expanded by checking and checking to obtain a horizontal line binary image.

[0076] In one embodiment, the step S3 comprises the following steps:

[0077] In step S31, the target coordinates are set to (X0, Y0), (X1, Y1), (X2, Y2), and (X3, Y3), and the two horizontal lines with the minimum and maximum y coordinates of the horizontal line binary image and the width not less than 0.5*img_w and the two vertical lines with the minimum and maximum x coordinates of the vertical line binary image and the width not less than 0.5*img_h are obtained, and the intersection of the two horizontal lines and the two vertical lines is calculated to obtain the intersection coordinates (x0, y0), (x1, y1), (x2, y2), and (x3, y3). The intersection coordinates (x0, y0), (x1, y1), (x2, y2), (x3, y3) and the target coordinates (X0, Y0), (X1, Y1), (X2, Y2), and (X3, Y3) are calculated through perspective transformation to obtain the perspective matrix M, and the vertical line binary image, the horizontal line binary image, and the three-channel image are corrected through the perspective matrix M.

[0078] In one embodiment, the step S4 comprises the following steps:

[0079] In step S41, the vertical line binary image and the horizontal line binary image after image correction are multiplied by 0.5 respectively and then added together to form a grid binary image. The grid binary image is negated, and the negated grid binary image is corroded and expanded by detection and verification. The grid binary image after the corrosion and expansion operations is subjected to a contour search method to obtain closed cells.

[0080] In one embodiment, the step S4 further includes the following steps:

[0081] In step S42, cells with too large an area are filtered out when the area of the cell satisfies w*h < img_w*img_h*alpha_1; where w represents the width of the cell, h represents the height of the cell, img_w represents the width of the image, img_h represents the height of the image, and alpha_1 is a coefficient between 0 and 1.

[0082] In step S43, cells with too small an area are filtered out when the area of the cell satisfies w*h > img_w*img_h*alpha_2, where w represents the width of the cell, h represents the height of the cell, img_w represents the width of the image, img_h represents the height of the image, alpha_2 is a coefficient between 0 and 1, and 0 < alpha_2 < alpha_1.

[0083] In step S44, the minimum value CELL_MIN_H of all cell heights and the minimum value CELL_MIN_W of widths are obtained. All cells are sorted in ascending order of the top-left y coordinate cell_min_y. The difference between the y coordinates cell_min_y of adjacent cells is calculated. If cell_min_y1 - cell_min_y0 is less than half of CELL_MIN_H, the two cells are considered to be in the same row, and min(cell_min_y1 - cell_min_y0) is stored as the y coordinate of this row in the row coordinate list row_y.

[0084] In step S45, the bottom-right y coordinate cell_max_y of all cells is traversed. If the difference between the bottom-right y coordinate cell_max_y and all rows in the row coordinate list row_y is greater than half of CELL_MIN_H, then cell_max_y is updated to the row coordinate list row_y.

[0085] In step S46, the row with the most cells is found. The x coordinate cell_min_x of the top-left corner of each cell is stored as the coordinate of each column in the column coordinate list col_x. The bottom-right x coordinate cell_max_x of all cells is traversed. If the difference between the x coordinate cell_max_x and all columns in the column coordinate list col_x is greater than half of CELL_MIN_W, then cell_max_x is updated to the column coordinate list col_x.

[0086] In one embodiment, step S5 includes the following steps:

[0087] In step S51, the text box is matched with the cell to obtain an intersection-over-union ratio, wherein the IOU matching calculation method is inter(text, cell).area / text.area, wherein inter(text, cell).area is the area of ​​the intersection of the text box and the cell, and text.area is the area of ​​the text box; if the intersection-over-union ratio of the current text box and all cells is less than or equal to the set threshold T2, it is determined that the current text box is not in any cell, and the text box is cached in the no_match list; if the intersection-over-union ratio of the current text box and all cells is greater than or equal to (1-T2), it is determined that the text box belongs to this cell, and the text box will not be cut, and the text box is cached in the match list; if the intersection-over-union ratio of the current text box and all cells is greater than T2 and less than 1-T2, the intersection of the cell and the text box is cut out as a new text box, and the new text box is attributed to this cell; if the intersection-over-union ratio is less than 0.5, the new text box is stored in the match_cache list, otherwise it is stored in the match list.

[0088] In one embodiment, the step S5 further includes the following steps:

[0089] In step S52, all text boxes belonging to the same cell in the match, match_cache and no_match lists are traversed, and the text boxes are recognized using the OCR recognition model to obtain text results, and the text results are combined into a character string from top to bottom and from left to right.

[0090] In one embodiment, the step S5 further includes the following steps:

[0091] In step S53, the text boxes in the no_match list are traversed, and the coordinates of the upper left corner of the text box (text_min_x, text_min_y) are used as reference points, and compared with the column coordinate list col_x and the row coordinate list

[0092] row_y, if the coordinates of the upper left corner of a text box meet the following conditions at the same time: column coordinate list

[0093] If the i-th element in col_x is less than text_min_x and text_min_x is less than the i+1-th element in col_x; if the j-th element in the row coordinate list row_y is less than text_min_y and text_min_y is less than the j+1-th element in row_y, the coordinates of the upper left corner of the text box are updated to (col_x[i], row_y[j]);

[0094] In step S54, the text boxes in the no_match list are traversed, and the coordinates of the lower right corner of the text box (text_max_x, text_max_y) are used as reference points, and compared with the column coordinate list col_x and the row coordinate list

[0095] row_y, if the coordinates of the lower right corner of a text box meet the following conditions at the same time: column coordinate list

[0096] If the i-th element in col_x is less than text_max_x and text_max_x is less than the i+1-th element in col_x; if the j-th element in the row coordinate list row_y is less than text_max_y and text_max_y is less than the j+1-th element in row_y, the coordinates of the upper left corner of the text box are updated to (col_x[j+1], row_y[i+1]);

[0097] In step S55, if the text box coordinates exceed the row coordinate list row_y or the column coordinate list col_x, they are expanded outwards with the minimum row spacing and the minimum column spacing until they are covered;

[0098] In step S56, the text boxes with the same coordinates of the upper left corner and the lower right corner are merged, and the texts are merged into a character string from top to bottom and from left to right.

[0099] In one embodiment, the step S6 comprises the following steps:

[0100] In step S61, the upper left corner coordinates (min_x, min_y) of the text boxes in the cells and the no_match list are traversed, and the upper left corner coordinates (min_x, min_y) are compared with the column coordinate list col_x and the row coordinate list row_y. If the following conditions are met at the same time: the absolute difference between min_y and the j-th element in the row coordinate list row_y is less than half of CELL_MIN_H; the absolute difference between min_x and the i-th element in the column coordinate list col_x is less than half of CELL_MIN_W, then an entry is added to the logical structure DICT_L to indicate that the text box is located at the i-th row and j-th column.

[0101] In step S62, the lower right corner coordinates (max_x, max_y) of the text box in the cell and no_match list are traversed, and compared with the column coordinate list col_x and the row coordinate list row_y according to the lower right corner coordinates (max_x, max_y). If the following conditions are met at the same time: the absolute difference between max_y and the mth element in the row coordinate list row_y is less than half of CELL_MIN_H; the absolute difference between max_x and the nth element in the column coordinate list col_x is less than half of CELL_MIN_W; then the position information of the text box is updated in the logical structure DICT_L, and the row and column where it is located and the number of cells (nj) and the number of cells (mi) that the text box needs to merge to the right and merge to the bottom are recorded;

[0102] In step S63, create a two-dimensional matrix M of size len(row_y)*len(col_x), the two-dimensional matrix M is used to record whether each cell has been filled, and initialize all elements to 0, indicating that no cell has been filled, traverse the logical structure DICT_L, and update the matrix M according to the position information of each text box; if a cell is covered in the previous step, set the corresponding matrix element to 1;

[0103] In step S64, the logical structure DICT_L is arranged in order from top to bottom by row i and from left to right by column j. The sorted logical structure DICT_L is generated into an HTML file, and the generated HTML file is read using the read_html function of pandas and converted into an Excel file.

[0104] Let the horizontal binary image be img1, the vertical binary image be img2, and the intersection of the horizontal binary image and the vertical binary image be represented as Where 0 represents the intersection point and t is the threshold;

[0105] Perspective transformation is to project a two-dimensional image onto a three-dimensional viewing plane, and then convert it to two-dimensional coordinates, so it is also called projection mapping. Simply put, it is a process of two-dimensional → three-dimensional → two-dimensional. The perspective transformation formula is as follows:

[0106]

[0107] In order to solve the eight parameters of the perspective transformation matrix, at least eight equations are required. Therefore, we need to select at least four sets of points (xi,yi) and (Xi,Yi) during calibration, representing the original image coordinates and the target image (rectified image) coordinates, assuming that they are (x0,y0) and (X0,Y0), (x1,y1) and (X1,Y1), (x2,y2) and (X2,Y2), (x3,y3) and (X3,Y3).

[0108] In this application, xi and yi are the coordinates of the four intersection points, starting from the upper left corner and arranged counterclockwise. Xi and Yi are (x0, y0), (x0, y0+H), (x0+W, y0+H), (x0+W, y0). The perspective matrix M is calculated through the four sets of coordinates.

[0109]

[0110] For the y coordinate of the lower right corner of all cells (let's call it cell_max_y), we need to check the difference between this coordinate and each recorded y coordinate in the row coordinate list row_y. Specifically, if the absolute difference between cell_max_y and each y coordinate in the row coordinate list row_y is greater than half of CELL_MIN_H, then we consider cell_max_y to be unique enough and should be added to the row coordinate list row_y.

[0111] In simple terms, it is to compare the distance between the y coordinate of the lower right corner of each cell and the existing row coordinates to decide whether to add the new y coordinate to the list as a new row. This process ensures that the new coordinate will only be added when it keeps a certain distance from all existing coordinates, thus avoiding duplicate or too close row coordinates from being added to the row coordinate list row_y.

[0112] For each text box in the no_match list, we use the coordinates of its upper left corner (text_min_x, text_min_y) as the reference point and compare the reference point with the predefined column coordinate list col_x and row coordinate list row_y. If the coordinates of the upper left corner of a text box meet the following conditions at the same time: the i-th element in the column coordinate list col_x is less than text_min_x and text_min_x is less than the i+1-th element in col_x; the j-th element in the row coordinate list row_y is less than text_min_y and text_min_y is less than the j+1-th element in the coordinate list row_y; then, the coordinates of the upper left corner of the text box will be updated to (col_x[i], row_y[j]).

[0113] In short, this process is to align the position of each text box in the no_match list to the nearest grid line (the grid defined by the column coordinate list col_x and the row coordinate list row_y), ensuring that the position of the text box is more regular and conforms to the preset row and column layout.

[0114] Traverse the upper left corner coordinates (min_x, min_y) of each text box in the cell and no_match list, and compare the upper left corner coordinates (min_x, min_y) with the pre-defined column coordinate list col_x and row coordinate list row_y. If the following conditions are met: the absolute difference between the j-th element in the column coordinate list col_x and the row coordinate list row_y is less than half of CELL_MIN_H; the absolute difference between the i-th element in the row coordinate list row_y and the column coordinate list col_x is less than half of CELL_MIN_W; then, add an entry to the logical structure DICT_L to indicate that this text box is located at the i-th row and j-th column. Specifically, DICT_L will be updated in the following format: {i:{j:()}}, where i represents the row number, j represents the column number, and brackets () can be used to store more information about the text box, such as its original coordinates or content.

[0115] The main purpose of this step is to build a logical structure based on rows and columns to better organize and manage the location information of text boxes, making subsequent processing (such as text extraction, typesetting analysis, etc.) more convenient and efficient.

[0116] Open the file and write HTML content by using with open(filename,"w",encoding="utf-8")as f:. Traverse each row and column to check whether the current cell has been processed. If the current cell exists in sorted_DICT_L, get its rowspan and colspan values ​​and write the corresponding HTML tags. Mark the processed cells to avoid repeated processing. If the current cell is not in sorted_DICT_L, write an empty tag. After running the code, an HTML file named output.html will be generated, which contains the merged cells arranged in order and writes the string str to the corresponding cell value. You can modify the value of the value variable as needed to write different strings.

[0117] This application uses dynamic thresholds in multiple places to improve the accuracy of recognition. Use the detection results of OCR to provide a reference threshold for the graphics algorithm. By using the IOU formula inter(text, cell).area / text.area; use cells to cut text boxes. Combine the coordinates of the upper left corner and the lower right corner of the cell to calculate the atomic table rows and columns (cannot be divided again) to solve the problem of special-shaped tables. The atomic table can be expanded according to the position of the text box. Create a matrix of the same scale as the atomic table rows and columns to store the status of whether a cell is occupied. This makes the application run fast, stable, and interpretable. Innovative, it has good handling of special-shaped tables, cross-page breaks, text cross-lines, text without wireless boxes, etc. It can be extended and has good compatibility with different types of wired table recognition. This application can be deployed on servers, personal PCs, and embedded devices, and has strong practical applicability.

[0118] The beneficial effects of a morphology-based wired table recognition method provided by an embodiment of the present invention are at least:

[0119] 1. This application can quickly convert the text in the picture into editable text, avoiding the tedious process of manual input and greatly improving work efficiency.

[0120] 2. By converting paper documents into electronic texts, paper usage and storage space can be reduced, reducing the company's operating costs.

[0121] 3. This application can avoid human input errors and improve the accuracy and reliability of data.

[0122] 4. After converting the text in the picture into electronic text, it can be easily shared and transmitted between different systems and platforms, promoting the circulation and utilization of information.

[0123] 5. This application can help enterprises and organizations automate and digitize their business processes.

[0124] 6. This application reduces costs for enterprises and improves work efficiency for individuals by directly converting images into Excel documents.

[0125] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should know that the present invention is not limited by the described action sequence, because according to the present invention, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.

[0126] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0127] Therefore, the above is only a preferred specific implementation mode of the present invention, and the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention. The protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A wired table recognition method based on morphology, characterized in that: The method comprises the following steps: S1, grayscale processing is performed on all the three-channel images to be detected to obtain grayscale images, and the obtained grayscale images are binarized to obtain binary images of the original images; S2, performing erosion and expansion operations on the binary image through a detection core, and detecting the binary image after the erosion and expansion operations to obtain a table line binary image; S3, performing image correction on the table line binary image and the three-channel image through a perspective matrix M; S4, combining the table line binary images after image correction into a grid binary image, negating the grid binary image, and performing erosion and expansion operations on the negated grid binary image by detection and verification, using a contour search method to obtain closed cells on the grid binary image after the erosion and expansion operations, filtering out cells with too large or too small areas and cells included by other cells, and calculating the rows and columns of the list where the cells are located; S5, using the OCR detection model to detect the text boxes in all the corrected three-channel images, performing IOU matching calculations on the text boxes and the cells, and determining whether the text boxes and the cells are successfully matched. If successful, the text boxes are cropped to obtain new text boxes, and the new text boxes are stored in a list. The OCR recognition model is used to perform text recognition on the text boxes in the list to obtain text results, and the text results are combined into strings from top to bottom and from left to right, and the rows and columns of the list are located according to the obtained text results. S6, recovering the table line binary image and character string to obtain an excel file.

2. The morphology-based wired table recognition method according to claim 1, characterized in that: The step S1 comprises the following steps: S11, setting a pixel grayscale threshold T1, and using the threshold T1 to convert a 0-255 grayscale image into a binary image.

3. The morphology-based wired table recognition method according to claim 2, characterized in that: The step S2 comprises the following steps: S21, performing corrosion and expansion operations on the binary image by detecting the core to obtain a vertical line binary image; S22, performing corrosion and expansion operations on the binary image by detecting and checking the binary image to obtain a horizontal line binary image.

4. The morphology-based wired table recognition method according to claim 3, characterized in that: The step S3 comprises the following steps: S31, set the target coordinates to (X0, Y0), (X1, Y1), (X2, Y2), (X3, Y3), and obtain the two horizontal lines with the minimum and maximum y coordinates of the horizontal line binary image and the width not less than 0.5*img_w and the two vertical lines with the minimum and maximum x coordinates of the vertical line binary image and the width not less than 0.5*img_h, and calculate the intersection of the two horizontal lines and the two vertical lines to obtain the intersection coordinates (x0, y0), (x1, y1), (x2, y2), (x3, y3), calculate the intersection coordinates (x0, y0), (x1, y1), (x2, y2), (x3, y3) and the target coordinates (X0, Y0), (X1, Y1), (X2, Y2), (X3, Y3) through perspective transformation to obtain the perspective matrix M, and use the perspective matrix M to perform image correction on the vertical line binary image, the horizontal line binary image and the three-channel image.

5. The morphology-based wired table recognition method according to claim 4, characterized in that: The step S4 comprises the following steps: S41. Multiply the vertical binary image and the horizontal binary image after image correction by 0.5 respectively, then add them together to form a grid binary image. Invert the grid binary image, and perform erosion and dilation operations on the inverted grid binary image through detection and verification. Use the contour finding method on the grid binary image after erosion and dilation operations to obtain closed cells.

6. The morphology-based wired table recognition method according to claim 5, characterized in that: The step S4 further includes the following steps: S42. Filter out cells with too large an area when the area of the cell satisfies w*h < img_w*img_h*alpha_1; where w represents the width of the cell, h represents the height of the cell, img_w represents the width of the image, img_h represents the height of the image, and alpha_1 is a coefficient between 0 and 1. S43. Filter out cells with too small an area when the area of the cell satisfies w*h > img_w*img_h*alpha_2, where w represents the width of the cell, h represents the height of the cell, img_w represents the width of the image, img_h represents the height of the image, alpha_2 is a coefficient between 0 and 1, and 0 < alpha_2 < alpha_1. S44. Obtain the minimum value CELL_MIN_H of the heights of all cells and the minimum value CELL_MIN_W of the widths. Sort all cells in ascending order according to the upper left corner y coordinate cell_min_y. Calculate the difference between the y coordinates cell_min_y of adjacent cells. If cell_min_y1 - cell_min_y0 is less than half of CELL_MIN_H, it is considered that the two cells belong to the same row, and min(cell_min_y1, cell_min_y0) is stored as the y coordinate of this row in the row coordinate list row_y. S45. Traverse the lower right corner y coordinate cell_max_y of all cells. If the difference between the lower right corner y coordinate cell_max_y and all rows in the row coordinate list row_y is greater than half of CELL_MIN_H, then update cell_max_y to the row coordinate list row_y. S46. Find the row with the most cells. Store the x coordinate cell_min_x of the upper left corner of each cell as the coordinate of each column in the column coordinate list col_x. Traverse the lower right corner x coordinate cell_max_x of all cells. If the difference between the x coordinate cell_max_x and all columns in the column coordinate list col_x is greater than half of CELL_MIN_W, then update cell_max_x to the column coordinate list col_x.

7. The morphology-based wired table recognition method according to claim 6, characterized in that: The step S5 includes the following steps: S51, match the text box with the cell to obtain the intersection-over-union ratio, wherein the IOU matching calculation method is inter(text, cell).area / text.area, wherein inter(text, cell).area is the area of ​​the intersection of the text box and the cell, and text.area is the area of ​​the text box; if the intersection-over-union ratio of the current text box and all cells is less than or equal to the set threshold T2, it is determined that the current text box is not in any cell, and the text box is cached in the no_match list; if the intersection-over-union ratio of the current text box and all cells is greater than or equal to (1-T2), it is determined that the text box belongs to this cell, and the text box will not be cut, and the text box is cached in the match list; if the intersection-over-union ratio of the current text box and all cells is greater than T2 and less than 1-T2, the intersection of the cell and the text box is cut out as a new text box, and the new text box is attributed to this cell. If the intersection-over-union ratio is less than 0.5, the new text box is stored in the match_cache list, otherwise it is stored in the match list.

8. The morphology-based wired table recognition method according to claim 7, characterized in that: The step S5 further comprises the following steps: S52, traverse all text boxes belonging to the same cell in the match and match_cache lists, use the OCR recognition model to perform text recognition on the text boxes to obtain text results, and combine the text results into a string from top to bottom and from left to right.

9. The morphology-based wired table recognition method according to claim 8, characterized in that: The step S5 further comprises the following steps: S53, traverse the text boxes in the no_match list, take the upper left corner coordinates of the text box (text_min_x, text_min_y) as the reference point, and compare with the column coordinate list col_x and the row coordinate list row_y. If the upper left corner coordinates of a text box simultaneously meet the following conditions: the i-th element in the column coordinate list col_x is less than text_min_x and text_min_x is less than the i+1-th element in col_x; the j-th element in the row coordinate list row_y is less than text_min_y and text_min_y is less than the j+1-th element in row_y, then the upper left corner coordinates of the text box are updated to (col_x[i], row_y[j]); S54, traverse the text boxes in the no_match list, take the coordinates of the lower right corner of the text box (text_max_x, text_max_y) as a reference point, and compare with the column coordinate list col_x and the row coordinate list row_y. If the coordinates of the lower right corner of a text box simultaneously meet the following conditions: the i-th element in the column coordinate list col_x is less than text_max_x and text_max_x is less than the i+1-th element in col_x; the j-th element in the row coordinate list row_y is less than text_max_y and text_max_y is less than the j+1-th element in row_y, then the coordinates of the upper left corner of the text box are updated to (col_x[j+1], row_y[i+1]); S55, if the coordinates of the text box exceed the row coordinate list row_y or the column coordinate list col_x, expand outwards with the minimum row spacing and the minimum column spacing until the coordinates of the text box are covered; S56, merge the text boxes with the same coordinates of the upper left corner and the lower right corner, and merge the text into a string from top to bottom and from left to right.

10. The morphology-based wired table recognition method according to claim 9, characterized in that: The step S6 comprises the following steps: S61, traverse the upper left corner coordinates (min_x, min_y) of the text box in the cell and no_match list, and compare the upper left corner coordinates (min_x, min_y) with the column coordinate list col_x and the row coordinate list row_y. If the following conditions are met at the same time: the absolute difference between min_y and the j-th element in the row coordinate list row_y is less than half of CELL_MIN_H; the absolute difference between min_x and the i-th element in the column coordinate list col_x is less than half of CELL_MIN_W, then add an entry to the logical structure DICT_L to indicate that the text box is located at the i-th row and j-th column. S62, traverse the lower right corner coordinates (max_x, max_y) of the text box in the cell and no_match list, and compare the lower right corner coordinates (max_x, max_y) with the column coordinate list col_x and the row coordinate list row_y. If the following conditions are met at the same time: the absolute difference between max_y and the mth element in the row coordinate list row_y is less than half of CELL_MIN_H; the absolute difference between max_x and the nth element in the column coordinate list col_x is less than half of CELL_MIN_W; then update the position information of the text box in the logical structure DICT_L, and record the row and column where it is located and the number of cells (nj) that the text box needs to merge to the right and the number of cells (mi) that the text box needs to merge to the bottom; S63, create a two-dimensional matrix M of size len(row_y)*len(col_x), the two-dimensional matrix M is used to record whether each cell has been filled, and initialize all elements to 0, indicating that no cell is filled, traverse the logical structure DICT_L, and update the matrix M according to the position information of each text box; if a cell is covered in the previous step, set the corresponding matrix element to 1; S64, arrange the logical structure DICT_L in order from top to bottom by row i and from left to right by column j, generate an HTML file from the sorted logical structure DICT_L, use the read_html function of pandas to read the generated HTML file and convert it into an Excel file.

Images