A deep learning-based table extraction and recognition method and system

Abstract

The application discloses a table extraction and recognition method and system based on deep learning, which comprises the following steps: S1, using a target detection model to locate the position of a table in a document picture, removing a repeated detection frame, obtaining the coordinates of the table, and cutting out a table picture from the original picture at the coordinates; S2, using an image segmentation model to detect table lines, obtaining the head and tail coordinates of the detected table lines, filling in the lines that are not fully detected, completing the lines that are not closed at the table boundary, obtaining the cell coordinates built by the table lines by marking the eight-connected regions of the binary image, and removing the cells that overlap and do not meet the requirements; S3, sorting all horizontal lines and vertical lines, obtaining the row and column distribution of the cells, reconstructing the table structure, performing OCR recognition on the original picture, making the text recognition content and the cells of the table one-to-one corresponding, and reconstructing the table content. The application improves the accuracy of table extraction and recognition.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to a table extraction and recognition method and system based on deep learning. BACKGROUND

[0002] With the rapid development of computer technology, the demand for converting table document images into text files is increasing. In the government field, it is necessary to frequently and massively audit and input table materials. If the table on the image can be intelligently recognized and the rows and columns of each cell can be standardized output, it can assist the government window service personnel to improve work efficiency, and also help users to realize the pre-filling function in the process of handling.

[0003] At present, most of the table recognition on the market is based on rules and layout alignment methods, which has the problems of low accuracy and poor universality.

[0004] Therefore, how to provide an effective scheme to accurately recognize the picture table is a problem to be solved in the prior art. SUMMARY

[0005] The purpose of the present application is to provide a table extraction and recognition method and system based on deep learning to solve the problems raised in the above technical background.

[0006] To achieve the above purpose, the present application adopts the following technical solutions:

[0007] The first aspect of the present application provides a table extraction and recognition method based on deep learning, comprising:

[0008] S1, table detection: using a target detection model to locate the position of the table in the document picture; removing the repeated detection frame, obtaining the coordinates of the table, and cutting out the table picture from the original picture with the coordinates;

[0009] S2, detecting table lines and obtaining cells:

[0010] S21, using an image segmentation model to detect table lines, and the target area predicted by the image segmentation model is a polygon area;

[0011] S22, marking the eight-connected region of the binary image of the predicted target area to obtain the contour point set coordinates of the target area, and calculating the minimum circumscribed rectangle of each target area, and taking the top left corner and the bottom right corner coordinates of the minimum circumscribed rectangle as the start and end coordinates of the table line;

[0012] S23, filling the incomplete lines in the table line detection;

[0013] S24, completing the line that is not closed in the table boundary;

[0014] S25, eight-connected region labeling is performed on the binarization image generated by the processed table lines to obtain cell coordinates built by all the table lines;

[0015] S26, overlapping cells and cells with sizes not meeting preset requirements are removed;

[0016] S3, table reconstruction: all horizontal lines and vertical lines in the table lines are sorted to obtain row and column distribution of cells, reconstruction of the table structure is realized, and OCR (Optical Character Recognition) is performed on the original image to recognize text recognition content and cells of the table one by one, and reconstruction of the table content is realized.

[0017] Preferably, step S1 specifically comprises the following steps:

[0018] S11, a yolov5 model is used to locate the table position;

[0019] S12, a non-maximum suppression (NMS) algorithm is used to remove repeated detection boxes of candidate boxes predicted by the yolov5 model to obtain specific coordinates of the table.

[0020] More preferably, step S12 specifically comprises the following steps:

[0021] The area size of n candidate boxes is calculated;

[0022] The confidence is sorted in descending order to obtain the subscript sequence after sorting, that is, argsort is used;

[0023] The candidate box with the maximum current confidence is added to the result list;

[0024] The intersection area of the candidate box with the maximum current confidence and any other candidate box is calculated;

[0025] The intersection-over-union of the candidate box is calculated using the intersection area and the area of the two candidate boxes, and the candidate box with an intersection-over-union greater than a preset threshold is deleted;

[0026] The above process is repeated for the remaining candidate boxes until all candidate boxes are processed.

[0027] Preferably, step S23 specifically comprises the following steps:

[0028] At least one target line segment in the table line is recognized, and a reference line is determined, the direction of the straight line where the target line segment is located is perpendicular to the direction of the straight line where the reference line is located;

[0029] The coordinates of the start and end points of the target line segment are p1(x1, y1) and p2(x2, y2), and the straight line equation of the reference line is Ax + By + C = 0, wherein A = y2 - y1, B = x2 - x1, and C = x2*y1 - x1*y2; the intersection point of the straight line where the target line segment is located and the straight line where the reference line is located is calculated:

[0030] Ax1 + By1 + C1 = 0, Ax2 + By2 + C2 = 0

[0031] x = (B1*C2 - B2*C1) / (A1*B2 - A2*B1)

[0032] y = (A2*C1 - A1*C2) / (A1*B2 - A2*B1)

[0033] Wherein (x, y) is the intersection point coordinates of the two straight lines;

[0034] The distances from the start and end points of the target line segment to the intersection point are calculated, and it is judged whether the minimum distance of the two calculated distances is greater than the preset distance threshold. If it is greater, the target line segment needs to be completed.

[0035] Preferably, step S24 specifically comprises the following steps:

[0036] All horizontal and vertical lines in the table are traversed, and the upper, lower, left and right four line segments at the boundary of the table area are obtained;

[0037] The positional relationship between the four line segments is judged, whether the line segments intersect each other is judged, the starting coordinates and the ending coordinates of each line segment are calculated respectively, and eight boundary coordinates are obtained;

[0038] According to the eight boundary coordinates, the straight line equations of the upper horizontal line, the lower horizontal line, the left vertical line and the right vertical line of the boundary of the table area are calculated respectively;

[0039] According to the four straight lines of the boundary, the intersection point coordinates of the four straight lines are calculated, and the boundary line of the table area is completed according to the intersection point coordinates.

[0040] Preferably, in step S25, the cell coordinates built by all table lines are obtained, specifically comprising the following steps:

[0041] The binary image is labeled with eight connected regions, and all polygon regions of the label are obtained;

[0042] The minimum circumscribed rectangle of the polygon region is obtained, and the coordinates of the upper left corner, the upper right corner, the lower left corner and the lower right corner are obtained;

[0043] The width and height of the minimum circumscribed rectangle are calculated according to the four point coordinates, and the calculation formula is as follows:

[0044] The coordinate values of a frame are known as (x1, y1, x1, y2, x3, y3, x4, y4), the mean value of all horizontal coordinates is calculated as c x =(x1+x2+x3+x4) / 4.0, the mean value of all vertical coordinates is calculated as c y =(y1+y2+y3+y4) / 4.0, then

[0045]

[0046] wherein w is the width of the minimum circumscribed rectangle, and h is the height of the minimum circumscribed rectangle;

[0047] The cell whose area is less than half of the area of the table region is reserved.

[0048] Preferably, the step S26 specifically comprises the following steps:

[0049] The intersection-over-union (IOU) of a cell and other cells is calculated, if the intersection-over-union is greater than a first preset multiple of the minimum area between them, the cell with smaller area is reserved, and the other cell is discarded.

[0050] The width and height of each cell are calculated according to the coordinates of all cells, the widths of all cells are sorted, the maximum and minimum values are removed, the mean value of the widths of all remaining cells is calculated, the heights of all cells are sorted, the maximum and minimum values are removed, and the mean value of the heights of all remaining cells is calculated; then all cells are traversed, if the width of the cell is less than a first preset proportion of the mean value of the width, and / or the height of the cell is less than a second preset proportion of the mean value of the height, the cell is determined as an irregular frame and needs to be deleted.

[0051] Preferably, the step S3 specifically comprises the following steps:

[0052] S31, all horizontal lines and vertical lines of the table are sorted, the intersection points (0, y) of all horizontal lines and the straight line x=0 are calculated, the size relationship of the y coordinates of the intersection points (0, y) is compared, all horizontal lines are sorted in up-down relationship, the intersection points (x, 0) of all vertical lines and the straight line y=0 are calculated, the size relationship of the x coordinates of the intersection points (x, 0) is compared, and all vertical lines are sorted in left-right relationship.

[0053] S32, the row and column distribution of each cell is sorted according to the table lines:

[0054] For each cell, the straight line equation of each horizontal line is calculated, and then the distances from the left upper corner point and the right lower corner point of the current cell to the straight line are calculated respectively, all horizontal lines are traversed in turn, and the two horizontal lines with the smallest distances from the left upper corner point and the right lower corner point respectively are obtained, which are the row distribution information of the cell.

[0055] For each cell, the straight line equation of each vertical line is calculated, and then the distance between the top-left corner and the bottom-right corner of the current cell and the straight line is calculated respectively. After traversing all the vertical lines, the two vertical lines with the smallest distance to the top-left corner and the bottom-right corner respectively are obtained, which are the column distribution information of the cell.

[0056] S33, performing OCR recognition on the original drawing to obtain all text content information and text box coordinate information. For each text box, all cells are traversed in turn to calculate the intersection-over-union (IOU) between the current text box and the cell. The calculation steps are as follows:

[0057] When both rectangles are positive rectangles, the coordinates of the two rectangles are rect1=(x1, y1, x2, y2) and rect2=(x3, y3, x4, y4). Calculate x min =max(x1, x3), y min =max(y1, y3), x max =min(x2, x4), y max =min(y2, y4), then IOU=(x max -x min )*(y max -y min ).

[0058] If the intersection area is greater than the second preset multiple of the text box area, the current text content information belongs to the cell. Thus, all text content information and all cells are one-to-one corresponding, and the table content restoration is completed.

[0059] The second aspect of the present application provides a deep learning-based table extraction and recognition system, comprising:

[0060] A table detection module is used to locate the position of the table in the document picture using a target detection model; remove duplicate detection boxes, obtain the coordinates of the table, and cut out the table picture from the original drawing at the coordinates;

[0061] A table line detection and cell acquisition module is used to detect table lines using an image segmentation model, predict a plurality of target regions with a polygon shape, perform eight-connected region labeling on the binary image of the predicted target region, obtain the contour point set coordinates of the target region, calculate the minimum circumscribed rectangle of each target region, and use the top-left corner and bottom-right corner coordinates of the minimum circumscribed rectangle as the start and end coordinates of the table line; fill in the incomplete lines of the table line detection; complete the lines of the table boundary that are not closed; perform eight-connected region labeling on the binary image generated by the processed table line to obtain the cell coordinates formed by all table lines; remove overlapping cells and small boxes that do not meet the preset requirements;

[0062] A table reconstruction module is configured to sort all horizontal and vertical lines in a table line, obtain row and column distribution of cells, reconstruct a table structure, perform OCR recognition on an original image, and correspond text recognition content and cells of the table to reconstruct table content.

[0063] The third aspect of the present application discloses a computer readable storage medium, which stores computer instructions, and the computer instructions perform the steps of the deep learning-based table extraction and recognition method.

[0064] Compared with the prior art, the technical scheme of the present application has the following beneficial effects:

[0065] The method of the present application has two functions that can be supported by wired table recognition. One is to determine the position of a cell according to row and column distribution information of the cell and extract a corresponding target field. The other is to restore a table document image to a real excel table to realize intelligent input of table text information.

[0066] The present application improves the accuracy of table extraction and recognition by using image processing and deep learning models, and has been applied in intelligent pre-examination and electronic material storage statistics in the government field. BRIEF DESCRIPTION OF DRAWINGS

[0067] The accompanying drawings, which form a part of this application, are included to provide a further understanding of the application and are incorporated in and constitute a part of this application. The embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0068] Figure 1 is a flowchart of a deep learning-based table extraction and recognition method of the present application;

[0069] Figure 2 is an example diagram of positioning a table position by using a yolov5 model in step S1 of the preferred embodiment of the present application;

[0070] Figure 3 is an example diagram of obtaining a final target detection frame after NMS processing in step S1 of the preferred embodiment of the present application, and the table image is cut out from the original document image;

[0071] Figure 4 (a) is a schematic diagram of a four-connected region in the embodiment of the present application;

[0072] Figure 4 (b) is a schematic diagram of an eight-connected region in the embodiment of the present application;

[0073] Figure 5 (a) is an example diagram of the detection position of a vertical line in a binary image in step S22 of the embodiment of the present application;

[0074] Figure 5 (b) is an example diagram of the target position marked with different colors for vertical lines in the binary image in step S22 of the embodiment of the present application;

[0075] Figure 6 is an example diagram of the incomplete detection of the table line in the embodiment of the present application, and the incomplete detection position is the upper right corner of the circled part;

[0076] Figure 7 is an example diagram of the incomplete closure of the table boundary in the embodiment of the present application, and the incomplete closure position is the lower circled part;

[0077] Figure 8 (a) is an example diagram of the binary table picture built for all table lines in step S25 of the embodiment of the present application;

[0078] Figure 8 (b) is an example diagram of the marking frame for marking each polygon with different colors in step S25 of the embodiment of the present application;

[0079] Figure 8 is an example diagram of the reduction of the table document picture to the real excel table to realize the intelligent input of the table text information by using the technical scheme of the present application. DETAILED DESCRIPTION

[0080] In order to make the purpose, technical scheme and effect of the present application more clear and explicit, the present application is further described in detail below with reference to the drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application.

[0081] It should be noted that the terms "first", "second" and the like in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not necessarily limit to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0082] EMBODIMENT

[0083] Figure 8 is a flowchart of a table extraction and recognition method based on deep learning.

[0084] As Figure 9As shown, a table extraction and recognition method based on deep learning, specifically comprising the following steps:

[0085] Figure 1

[0086] The specific steps are as follows:

[0087] S11, the first step of table recognition is to detect the position of the table in the document picture, and in this embodiment, the yolov5 model is used to locate the table position.

[0088] S12, there is a case where a target is detected multiple times in the candidate box predicted by the yolov5 model, such as Step S1: Table detection. In order to obtain an accurate detection box, a group of candidate boxes need to be subjected to NMS (Non-Maximum Suppression) operation. The implementation principle of NMS is to search for local maximum value and suppress non-maximum value elements, so as to eliminate some repeated values.

[0089] The process of NMS implementation is as follows:

[0090] (1) Calculate the area size of n candidate boxes;

[0091] (2) Sort the confidence in descending order to obtain the subscript sequence number, that is, use argsort;

[0092] (3) Add the candidate box with the maximum current confidence to the result list;

[0093] (4) Calculate the intersection area of the candidate box with the maximum current confidence and any other candidate box;

[0094] (5) Calculate the intersection-over-union of the two candidate boxes using the intersection area and the area of the two candidate boxes, and delete the candidate box with an intersection-over-union greater than the threshold;

[0095] (6) Repeat the above process for the remaining candidate boxes until all candidate boxes are processed.

[0096] S13, the final target detection box is obtained after NMS processing, and the table picture is cut out from the original document image according to the coordinates of the target detection box, such as Figure 2 .

[0097] Figure 3

[0098] The specific steps are as follows:

[0099] S21, the wired table is a structure built by crossing multiple horizontal and vertical lines, and before restoring the table structure, the table lines need to be detected first, and in this embodiment, the unet image segmentation model is used to detect the table lines.

[0100] S22, the target region predicted by the model is a polygon region in strict sense. In order to extract the start and end coordinates of the table line, the following steps are performed:

[0101] (1) eight-connected region labeling is performed on the predicted target region to obtain the contour point set coordinates of the predicted target region.

[0102] The predicted binary image has only two possibilities of 0 (black) and 255 (white), and the pixels of each target region in the binary image are 255, and these target pixels form a connected region in a certain distribution. The connected region refers to two pixel points adjacent to each other and having the same pixel, so that the two pixel points are in the same connected region. From the visual point of view, the points connected to each other form a region, and the set of all connected points in the region is called a connected region. When judging whether two points are adjacent, there are two judgment methods: four-connected and eight-connected. Four-connected considers four adjacent pixels, i.e. up, down, left and right, as shown in Step S2: Detect table lines, get cells. (a); while eight-connected considers eight adjacent pixels, i.e. up, down, left, right and diagonal pixels, as shown in Figure 4 (b).

[0103] After finding the connected region in the binary image, each connected region is marked with a different symbol, as shown in Figure 4 . Figure 5 The picture of (a) is the detection position of the vertical line in the binary image, Figure 5 The picture of (b) is the target position after marking.

[0104] (2) the minimum circumscribed rectangle of each target region is obtained, and the left upper corner and right lower corner coordinates of the rectangle are taken as the start and end coordinates of the table line.

[0105] Basic principle of minimum circumscribed rectangle:

[0106] The initial x min , y min , x max , y max of the point set are obtained to determine the initial circumscribed rectangle, the area of the rectangle is calculated, then the point set is rotated according to the following rotation formula:

[0107] x’ = xcos(angle) - ysin(angle)

[0108] y’ = xsin(angle) + ycos(angle)

[0109] The coordinates of the point set after rotation are calculated, and then the x′ min , y′ min , x′ max , y′ max, calculate the current rectangular area, until the area is the minimum, that is, the minimum circumscribed rectangle.

[0110] S23, when some lines in the table picture are very blurred, the table line detection is not complete, and the lines need to be filled, the steps are as follows:

[0111] (1) Calculate the straight line general equation of a target horizontal line segment (vertical line segment) of the table line and the reference vertical line (horizontal line), and the calculation formula is as follows:

[0112] The coordinates of the first and last points of the line segment are p1(x1, y1), p2(x2, y2), and the straight line general equation is Ax+By+C=0

[0113] A=y2-y1

[0114] B=x1-x2

[0115] C=x2*y1-x1*y2

[0116] (2) Determine the position relationship between the first and last points of the target horizontal line segment (vertical line segment) and the reference vertical line (horizontal line), and calculate as follows: r=A*x1+B*y1+C. If r=0, the point is on the straight line; otherwise, the point is outside the straight line.

[0117] (3) If the table line detection is not complete, such as Figure 5 The upper right corner of the circled part. If the first and last points of the target horizontal line segment (vertical line segment) are on the same side of the reference straight line, first calculate the intersection point of the straight line where the target horizontal line segment (vertical line segment) is located and the reference straight line, and the intersection point calculation formula is as follows:

[0118] The general equations of the two straight lines are:

[0119] A1x+B1y+C1=0, A2x+B2y+C2=0

[0120] x=(B1*C2-B2*C1) / (A1*B2-A2*B1)

[0121] y=(A2*C1-A1*C2) / (A1*B2-A2*B1)

[0122] Where (x, y) is the intersection point coordinates of the two straight lines.

[0123] Then calculate the distance from the first and last points of the target horizontal line segment (vertical line segment) to the intersection point, and determine whether the minimum distance between the two is greater than a given threshold. If it is greater, it means that the line segment needs to be completed.

[0124] S24, some pictures in the table will exist in the state of boundary not closed, such as Figure 6The boundary of the lowermost circle in the figure is not closed, so the table area needs to be completed.

[0125] The steps are as follows:

[0126] (1) Traverse all horizontal and vertical lines to find the upper and lower and left and right four line segments of the boundary.

[0127] (2) Determine the positional relationship between the four line segments, such as calculating the positions of the first and last points of the uppermost horizontal line and the left and right two vertical lines, respectively. If both points are on the same side of the two straight lines, the left coordinate of the uppermost is the initial coordinate of the horizontal line, and the right coordinate of the uppermost is the terminal coordinate of the horizontal line; otherwise, the left coordinate of the uppermost is the intersection coordinate of the horizontal line and the left vertical line, and the right coordinate of the uppermost is the intersection coordinate of the horizontal line and the right vertical line. Eight boundary coordinates are calculated in this way.

[0128] (3) According to the eight boundary coordinates, the straight line equations of the upper horizontal line, the lower horizontal line, the left vertical line, and the right vertical line of the region boundary are calculated.

[0129] (4) According to the four straight lines of the boundary, the intersection coordinates of the four straight lines are calculated, and the table boundary is completed according to these intersection coordinates.

[0130] S25, in order to find the cell in the table, the eight-connected region marking is done on the binary table picture built by all table lines, the positional attribute of the connected region is calculated, and the polygon frame formed by the table lines is obtained. Referring to Figure 7 , wherein Figure 8 (a) is an example of a binary table picture built by all table lines, Figure 8 (b) is an example of a marked frame, Figure 8 Each polygon in (b) is marked with a different color.

[0131] Wherein, the steps of obtaining the cell are as follows:

[0132] (1) Mark the eight-connected region on the binary picture, and find all the polygon regions marked.

[0133] (2) Find the minimum circumscribed rectangle of the polygon region, and obtain the coordinates of its top-left corner, top-right corner, bottom-left corner, and bottom-right corner.

[0134] (3) Calculate the width and height of the minimum circumscribed rectangle according to the four-point coordinates, and the calculation formula is as follows: Given the coordinate value of a frame as (x1, y1, x1, y2, x3, y3, x4, y4), calculate the mean value of all horizontal coordinates as c x = (x1+x2+x3+x4) / 4.0, and calculate the mean value of all vertical coordinates as c y = (y1+y2+y3+y4) / 4.0, then

[0135]

[0136]

[0137] wherein w is the width of the minimum circumscribed rectangle, and h is the height of the minimum circumscribed rectangle.

[0138] (4) A cell with a reserved area less than half the area of the table region.

[0139] S26, since the cell is located by finding the eight-connected region of the binary image, there may be overlapping boxes, which need to be processed for deduplication; or because the table line itself has a certain degree of inclination, it will cause some small boxes that do not meet the requirements, which need to be deleted. The specific steps are as follows:

[0140] (1) Calculate the intersection over union (IOU) of a cell with other cells. If the value is greater than 0.4 times the minimum area between them, the cell with the smaller area between them is retained, and the other cell is discarded.

[0141] (2) Calculate the width and height of the cell according to the coordinates of all cells, sort the width (height) of all cells, remove the maximum and minimum values, and calculate the mean value of the width (height) of all remaining cells. Then traverse all cells, if the width (height) of the cell is less than a certain proportion of the mean width (height), the cell is a very irregular box and needs to be removed.

[0142] Figure 8

[0143] The specific steps are as follows:

[0144] S31, sort all horizontal and vertical lines of the table, calculate the intersection point (0, y) of all horizontal lines and the straight line x = 0, compare the size relationship of the y coordinates in the intersection point (0, y), which can sort all horizontal lines in the up-down relationship, and take similar measures for all vertical lines.

[0145] S32, according to the table line, sort the row and column distribution of each cell, for each cell, calculate the straight line equation of each horizontal line, and then calculate the distance from the left upper corner point and the right lower corner point of the current cell to the straight line respectively, the calculation formula is as follows:

[0146] Given a point p(x, y) and a general equation of a straight line Ax + By + C = 0,

[0147] The distance from the point to the straight line is:

[0148] After traversing all the horizontal lines in turn, the two horizontal lines with the minimum distance from the upper left corner point and the lower right corner point are obtained, which are the row distribution information of the cell, and the column distribution information of the cell is calculated in the same way as above, so that the table structure is reconstructed.

[0149] S33, the original drawing is recognized by OCR to obtain all text content information and text box coordinate information, for each text box, all cells are traversed in turn, and the intersection over union IOU of the current text box and the cell is calculated respectively, the calculation steps are as follows:

[0150] When both rectangles are positive rectangles, the coordinates of the two rectangles are rect1=(x1,y1,x2,y2) and rect2=(x3,y3,x4,y4) respectively.

[0151] x min =max(x1,x3);

[0152] y min =max(y1,y3);

[0153] x max =min(x2,x4);

[0154] y max =min(y2,y4);

[0155] IOU=(x max -x min )*(y max -y min );

[0156] If the intersection area is greater than 0.5 times the area of the text box, the current text content information belongs to the cell, and all text content information and all cells are corresponded one by one, and the table content is restored.

[0157] The technical scheme of the present application has two functions that can support the current wired table recognition: one is to determine the position of the cell according to the row and column distribution information of the cell, and extract the corresponding target field; the other is to restore the table document picture to the real excel table, and realize the intelligent input of table text information, as shown in Step S3: Table reconstruction. Figure 9 .

[0158] On the other hand, the present application also provides a table extraction and recognition system based on deep learning, comprising:

[0159] A table detection module is used to locate the position of the table in the document picture using a target detection model, remove duplicate detection boxes, obtain the coordinates of the table, and cut out the table picture from the original picture at the coordinates.

[0160] The table line detection and cell obtaining module is configured to detect table lines using an image segmentation model, predict a plurality of target regions in the shape of polygonal regions, mark the binary image of the predicted target regions in eight-connected regions, obtain the contour point set coordinates of the target regions, calculate the minimum circumscribed rectangle of each target region, and take the upper left corner and lower right corner coordinates of the minimum circumscribed rectangle as the start and end coordinates of the table lines; fill in the lines that are not fully detected; complete the lines that are not closed; mark the binary image generated by the processed table lines in eight-connected regions, and obtain the cell coordinates formed by all the table lines; and remove the overlapping cells and small boxes that do not meet the preset requirements.

[0161] The table reconstruction module is configured to sort all the horizontal lines and vertical lines in the table lines, obtain the row and column distribution of the cells, reconstruct the table structure, perform OCR recognition on the original image, correspond the text recognition content to the cells of the table one by one, and reconstruct the table content.

[0162] In another aspect, the present application also provides a computer readable storage medium having computer instructions stored thereon, wherein the computer instructions perform the steps of the above-mentioned table extraction and recognition method based on deep learning when running.

[0163] In actual applications, the computer instructions in the embodiments of the present application can be written in one or more program design languages in any combination to program code for executing the operations of the embodiments of the present application, the program design languages including object-oriented program design languages such as Java, C++, and conventional procedural program design languages such as the "C" language or similar program design languages. The program code can be completely executed on a user computing device, partially executed on a user device, executed as a separate software package, partially executed on a user computing device and partially executed on a remote computing device, or completely executed on a remote computing device or server.

[0164] In actual applications, the computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium may, for example, include but is not limited to an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include an electrical connection having one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0165] Those skilled in the art will further appreciate that the various illustrative logical blocks, modules, and algorithm steps described in connection with the disclosure herein can be implemented as electronic hardware, computer software, or combinations of both.

[0166] In summary, the present application improves the accuracy of table extraction and recognition by using image processing and deep learning models, and has been applied in intelligent pre-trial and electronic material storage statistics in the government field.

[0167] The specific embodiments of the present application are described in detail above, but it is only as an example, and the present application is not limited to the specific embodiments described above. Any equivalent modification and substitution of the present application for those skilled in the art are also within the scope of the present application. Therefore, equivalent transformations and modifications made without departing from the spirit and scope of the present application should be covered within the scope of the present application.

Claims

1. A deep learning-based method for table extraction and recognition, characterized in that, include: S1, Table Detection: Use an object detection model to locate the position of tables in a document image; Remove duplicate detection boxes, obtain the coordinates of the table, and extract the table image from the original image using those coordinates; S2, detect table lines and retrieve cells: S21, Use an image segmentation model to detect table lines. The target region predicted by the image segmentation model is a polygonal region. S22, mark the binarized image of the predicted target region with eight connected regions, obtain the coordinates of the contour point set of the target region, find the minimum bounding rectangle of each target region, and use the coordinates of the upper left and lower right corners of the minimum bounding rectangle as the coordinates of the beginning and end of the table line. S23, fill in the missing lines in the table line detection; S24, complete the lines that are not closed at the table boundaries; S25, mark the octagonal connected regions of the binarized image generated by the processed table lines, and obtain the cell coordinates constructed by all the table lines; S26, remove overlapping cells and cells whose size does not meet the preset requirements; S3, Table Reconstruction: Sort all horizontal and vertical lines in the table to obtain the row and column distribution of the cells, thus reconstructing the table structure. Also, perform OCR recognition on the original image, matching the recognized text content with the table cells to reconstruct the table content. Specifically, this includes the following steps: S31. Sort all horizontal and vertical lines in the table, calculate the intersection point (0,y) of all horizontal lines with the line x=0, compare the size of the y coordinates at the intersection point (0,y), sort all horizontal lines by their vertical relationship, calculate the intersection point (x,0) of all vertical lines with the line y=0, compare the size of the x coordinates at the intersection point (x,0), and sort all vertical lines by their horizontal relationship. S32, organize the row and column distribution for each cell according to the table lines: For each cell, calculate the equation of the line containing each horizontal line, then calculate the distance from the top left and bottom right corners of the current cell to the line. After traversing all the horizontal lines, obtain the two horizontal lines with the smallest distances to the top left and bottom right corners, which are the row distribution information of the cell. For each cell, calculate the equation of the line containing each vertical line, then calculate the distance from the top left and bottom right corners of the current cell to the line. After traversing all the vertical lines, obtain the two vertical lines with the smallest distances to the top left and bottom right corners, which are the column distribution information of the cell. S33. Perform OCR recognition on the original image to obtain all text content information and text box coordinate information. For each text box, iterate through all cells in turn and calculate the Intersection over Union (IOU) between the current text box and the cell. The calculation steps are as follows: When both rectangles are regular rectangles, their coordinates are rect1=(x1, y1, x2, y2) and rect2=(x3, y3, x4, y4). Calculate x. min =max(x1, x3), y min =max(y1, y3), x max =min(x2, x4), y max =min(y2,y4), then IOU=( x max - x min )*( y max - y min ); If the intersection area is greater than a second preset multiple of the text box area, then the current text content belongs to that cell. This allows all text content to be matched with all cells, thus restoring the table content.

2. The table extraction and recognition method based on deep learning according to claim 1, characterized in that, Step S1 specifically includes the following steps: S11, Use the YOLOv5 model to locate the table position; S12, Non-maximum suppression (NMS) algorithm is used to remove duplicate detection boxes from the candidate boxes predicted by the YOLOv5 model to obtain the specific coordinates of the table.

3. The table extraction and recognition method based on deep learning according to claim 2, characterized in that, Step S12 specifically includes the following steps: Calculate the area of ​​n candidate boxes; Sort the confidence scores in descending order and obtain the sorted index numbers, i.e., use argsort; Add the candidate box with the highest confidence level to the results list; Calculate the intersection area of ​​the candidate box with the highest confidence score with any other candidate box; The intersection-union ratio (IUR) of candidate boxes is calculated using the intersection area and the area of ​​each candidate box itself, and candidate boxes with an IUR greater than a preset threshold are deleted. Repeat the above process for the remaining candidate boxes until all candidate boxes have been processed.

4. The deep learning-based table extraction and recognition method according to claim 1, characterized in that, Step S23 specifically includes the following steps: Identify at least one target line segment in the table lines and determine a reference line, wherein the direction of the line containing the target line segment is perpendicular to the direction of the line containing the reference line. Let the coordinates of the first and last points of the target line segment be p1 (x1, y1) and p2 (x2, y2) respectively. Let the equation of the reference line be Ax + By + C = 0, where A = y2 - y1, B = x2 - x1, and C = x2 * y1 - x1 * y2. Calculate the intersection point of the line containing the target line segment and the line containing the reference line: Ax1 + By1 + C1=0, Ax2 + By2 + C2=0 x = (B1*C2- B2*C1 ) / (A1*B2- A2*B1 ) y = (A2*C1- A1*C2 ) / (A1*B2- A2*B1 ) Where (x, y) are the coordinates of the intersection of the two lines; Calculate the distances from the coordinates of the first and last points of the target line segment to the intersection point. Determine whether the minimum distance among the two calculated distances is greater than a preset distance threshold. If it is greater, the target line segment needs to be completed.

5. The deep learning-based table extraction and recognition method according to claim 1, characterized in that, Step S24 specifically includes the following steps: Iterate through all the horizontal and vertical lines in the table and find the four line segments at the top, bottom, left, and right of the table area boundary. Determine the positional relationship between the four line segments, determine whether the line segments intersect, and calculate the starting and ending coordinates of each line segment to obtain a total of eight boundary coordinates; Based on these eight boundary coordinates, calculate the equations of the upper horizontal line, lower horizontal line, left vertical line, and right vertical line of the table area boundary, respectively; Based on the four straight lines of the boundary, calculate the coordinates of the intersection points of these four lines, and use these intersection point coordinates as a reference to complete the boundary lines of the table area.

6. The table extraction and recognition method based on deep learning according to claim 1, characterized in that, In step S25, obtaining the cell coordinates constructed by all table lines specifically includes the following steps: Mark the binarized image with 8-connected regions and find all the marked polygonal regions; Find the smallest bounding rectangle of the polygonal region and obtain the coordinates of its top-left, top-right, bottom-left, and bottom-right corners; The width and height of the minimum bounding rectangle can be calculated using the coordinates of four points, as follows: Given the coordinates of a box (x1, y1, x1, y2, x3, y3, x4, y4), calculate the mean of all x-coordinates and the mean of all y-coordinates. Where w is the width of the minimum bounding rectangle and h is the height of the minimum bounding rectangle; Retain cells whose area is less than half the area of ​​the table area.

7. The deep learning-based table extraction and recognition method according to claim 1, characterized in that, Step S26 specifically includes the following steps: Calculate the Intersection over Union (IOU) of a cell with other cells. If the IOU is greater than a first preset multiple of the smallest area between the two cells, keep the cell with the smaller area and discard the other cell. Calculate the width and height of each cell based on the coordinates of all cells. Sort all cell widths, remove the largest and smallest widths, and calculate the average width of all remaining cells. Sort all cell heights, remove the largest and smallest heights, and calculate the average height of all remaining cells. Then iterate through all cells. If a cell's width is less than a first preset percentage of the average width, and / or its height is less than a second preset percentage of the average height, then the cell is considered an irregular frame and needs to be deleted.

8. A deep learning-based table extraction and recognition system, characterized in that, include: Table detection module: Used to locate the position of tables in document images using an object detection model; Remove duplicate detection boxes, obtain the coordinates of the table, and extract the table image from the original image using those coordinates; The table line detection and cell acquisition module is used to detect table lines using an image segmentation model, predicting multiple target regions with polygonal shapes; it marks the binarized images of the predicted target regions with 8-connected regions, obtains the coordinates of the contour point set of the target regions, calculates the minimum bounding rectangle of each target region, and uses the coordinates of the top left and bottom right corners of the minimum bounding rectangle as the coordinates of the beginning and end of the table line; it fills in incomplete table line detection; it completes lines with unclosed table boundaries; it marks the binarized images generated from the processed table lines with 8-connected regions, obtains the coordinates of the cells constructed by all table lines; and it removes overlapping cells and small boxes whose sizes do not meet the preset requirements. Table Reconstruction Module: This module sorts all horizontal and vertical lines in a table, obtains the row and column distribution of cells, reconstructs the table structure, and performs OCR recognition on the original image, mapping the recognized text content to the table cells to reconstruct the table content. Specifically, it includes: Sort all horizontal and vertical lines in the table, calculate the intersection point (0,y) of all horizontal lines with the line x=0, compare the size of the y coordinates at the intersection point (0,y), sort all horizontal lines vertically, calculate the intersection point (x,0) of all vertical lines with the line y=0, compare the size of the x coordinates at the intersection point (x,0), and sort all vertical lines horizontally. Based on the table lines, analyze the row and column distribution of each cell: For each cell, calculate the equation of the line containing each horizontal line, then calculate the distance from the top-left and bottom-right corners of the current cell to that line. Repeat this process for all horizontal lines to obtain the two horizontal lines with the smallest distances to the top-left and bottom-right corners, which represent the row distribution information of that cell. For each cell, calculate the equation of the line containing each vertical line, then calculate the distance from the top-left and bottom-right corners of the current cell to that line. Repeat this process for all vertical lines to obtain the two vertical lines with the smallest distances to the top-left and bottom-right corners, which represent the column distribution information of that cell. Perform OCR recognition on the original image to obtain all text content information and the coordinate information of the text boxes. For each text box, iterate through all cells and calculate the Intersection over Union (IOU) between the current text box and the cell. The calculation steps are as follows: When both rectangles are regular rectangles, their coordinates are rect1=(x1, y1, x2, y2) and rect2=(x3, y3, x4, y4). Calculate x. min =max(x1, x3), y min =max(y1, y3), x max =min(x2, x4), y max =min(y2,y4), then IOU=( x max - x min )*( y max - y min ); If the intersection area is greater than a second preset multiple of the text box area, then the current text content belongs to that cell. This allows all text content to be matched with all cells, thus restoring the table content.

9. A computer-readable storage medium storing computer instructions thereon, characterized in that, When the computer instructions are executed, they perform the steps of the deep learning-based table extraction and recognition method according to any one of claims 1 to 7.

Images