Table structure detection method for flattening irregular table images

Abstract

This invention discloses a table structure detection method for flattening irregular table images, comprising: 1. detecting table regions in the original image to obtain table region images; 2. performing table regularity checks on the table region images; 3. preprocessing non-standard table region images; and 4. identifying borders in the table region images. This invention employs a novel method—a technique for preprocessing and flattening irregular tables to make them more regular. This method effectively solves the problem of table irregularity, thereby improving the algorithm's operability and extracting semantic information from irregular tables. Furthermore, compared to deep learning algorithms, this algorithm saves the time required for model training, significantly improving the time efficiency of table recognition.

Description

Technical Field

[0001] This invention belongs to the field of image detection technology, specifically relating to a table structure detection method for flattening irregular table images. Background Technology

[0002] OCR refers to the process by which electronic devices (such as scanners or digital cameras) examine printed characters on paper, determine their shapes by detecting dark and light patterns, and then translate those shapes into computer text using character recognition methods. This algorithm can convert digital images of scanned documents into machine-readable information. Table text recognition is a technique within OCR used to detect table images. This technology can be applied in fields such as education and finance. For example, it can parse student grades in images and automatically input them into a system. In the financial industry, it can parse data from paper reports and input it into the system. Table text recognition technology has significant practical implications for people's daily lives.

[0003] Table text recognition can be simply divided into two parts: one part is the detection of the table in the image. Currently, commonly used object detection algorithms are used to identify the position of the table in the image. These methods are quite effective in extracting the table borders. The other part is the recognition of the table structure. Current table recognition technologies include: (1) using image processing methods to segment the table lines; (2) using rules based on the text detection coordinates to reconstruct the table; (3) using deep learning methods to segment the table lines and using rules to reconstruct the table. This technology uses image processing methods to segment the table lines. Compared with using deep learning methods to detect the table structure, this is faster and more intuitive. It is also very effective in extracting the table structure in PDFs. However, for photos taken by users or some skewed tables, the effect of detecting the table structure will naturally become very poor.

[0004] Chinese patent CN110163198A discloses a method for table recognition and reconstruction. This method uses a table frame line model to segment the table, detects the row and column lines of the table from the segmented image to obtain their position information, and generates an electronic spreadsheet file in the corresponding format based on the position information of the table lines and the position information of the cells. However, its disadvantage is that for irregular table images, it is necessary to manually confirm whether the generated borders are reasonable. Furthermore, this patent technology is aimed at table reconstruction and cannot make a true table semantics of the table in the original image, so it cannot perform subsequent text recognition within the table.

[0005] Chinese patent CN106407883A discloses a method for recognizing handwritten digits in complex tables. It first preprocesses the complex table template, detects straight lines, sorts and classifies corner points by row and column, and defines the template table to describe the structure of the cells. After obtaining the electronic handwritten table, it corrects the tilt and adapts it to the template table to obtain the cell position description. Then, each cell is processed, removing borders while preserving the characters within the cell as completely as possible. Next, the digit images in the cells are extracted, and a classifier trained on the dataset is used to recognize the digit images. Finally, the handwritten characters are post-processed, and the recognition results are filled into the template table. However, its drawback is that the image preprocessing process only applies to paper handwritten table documents and does not preprocess photographs of tables. Summary of the Invention

[0006] In view of the above, the present invention provides a table structure detection method for flattening irregular table images, making the table regular. This method can effectively solve the problem of table irregularity, thereby improving the efficiency and operability of the algorithm.

[0007] A method for detecting table structure in irregular table images, comprising the following steps:

[0008] (1) Perform table region detection on the original image to obtain table region images;

[0009] (2) Perform a table standardization check on the image of the table area;

[0010] (3) Preprocess the images of table areas that do not conform to the specifications;

[0011] (4) Recognize the borders of the table area image.

[0012] Further, the specific implementation of step (1) is as follows: input the original image containing the table into the pre-trained deep learning target detection model to detect the regional location information of the table in the output image, and then extract the table to obtain a cropped table region image.

[0013] Furthermore, the specific implementation process of step (2) is as follows:

[0014] 2.1 Convert the table area image to grayscale, and then use adaptive thresholding to perform binarization and dilation operations on the grayscale image;

[0015] 2.2 Perform contour edge detection on the image processed in step 2.1, detect all the table borders in the image, and expand these borders.

[0016] 2.3 Based on the coordinate position information of each frame line, find the outline with the largest area in the image. This outline is the table outline.

[0017] 2.4 Determine whether the table is standardized based on the coordinate information of the four vertices of the table outline.

[0018] Furthermore, the standard for judging whether a table is standardized in step 2.4 is as follows: if all of the following conditions are met, the table is judged to be standardized; otherwise, the table is judged to be non-standard.

[0019] y1∈[y2-5,y2+5]&y3∈[y4-5,y4+5]

[0020] x1∈[x3-10,x3+10]&x2∈[x4-10,x4+10]

[0021] Wherein: the coordinates of the top left corner vertex of the table outline are (x1, y1), the coordinates of the top right corner vertex are (x2, y2), the coordinates of the bottom left corner vertex are (x3, y3), and the coordinates of the bottom right corner vertex are (x4, y4).

[0022] Furthermore, the specific implementation process of step (3) is as follows:

[0023] 3.1 Convert the non-compliant table area image to a grayscale image, and then use an adaptive threshold to perform binarization and dilation operations on the grayscale image;

[0024] 3.2 Perform contour edge detection on the image processed in step 3.1, detect all the table borders in the image, and expand these borders.

[0025] 3.3 Based on the coordinate position information of each frame line, find the outline with the largest area in the image. This outline is the table outline.

[0026] 3.4 For the four curves of the table outline, the curve functions of these four curves are fitted by the least squares method using the coordinates of the outline points obtained by edge detection.

[0027] 3.5 For any curve, subdivide it into multiple segments. Each segment can be regarded as a straight line. Use the curve function to calculate the coordinates of the first and last points of each straight line, and then calculate the length of each straight line. The sum of these segments is the length of the curve.

[0028] 3.6 Using the coordinate information of the four vertices of the table outline and the lengths of the four curves, the table area image is corrected through perspective transformation.

[0029] Furthermore, the specific implementation of step 3.6 is as follows: First, the table outline is divided into multiple blocks of 10×10. The coordinates of the four vertices of each block are calculated based on the coordinate information of the four vertices and the length of the four curves. Then, the two curves at the bottom and left of the table outline are used as the x-axis and y-axis in the new coordinate system, respectively. The coordinates of the four vertices of each block in the new coordinate system are calculated. Then, based on the coordinates of these points in the original coordinate system and the new coordinate system, the table area image is projected onto a new viewing plane through perspective transformation, thereby completing the correction.

[0030] Furthermore, the specific implementation process of step (4) is as follows:

[0031] 4.1 Convert the table area image to grayscale, then use adaptive thresholding to binarize the grayscale image, and perform image preprocessing;

[0032] 4.2 The preprocessed image is subjected to erosion and dilation processes in sequence to obtain an image with horizontal lines and an image with vertical lines;

[0033] 4.3 Combine the horizontal and vertical line images to obtain a complete raster unit image, and then perform erosion processing on the image.

[0034] 4.4 Finally, edge detection is used to obtain the coordinate data of each cell in the raster unit image, and a text file containing the information of each cell is generated.

[0035] This invention employs a novel method, a technique that preprocesses and flattens irregular tables to make them regular. This method effectively solves the problem of table irregularity, thereby improving the operability of the algorithm and extracting semantic information from irregular tables. Furthermore, compared to deep learning algorithms, this algorithm saves the time of training the model, greatly improving the time efficiency of table recognition. Attached Figure Description

[0036] Figure 1 This is a schematic diagram of the steps of the table structure detection method of the present invention.

[0037] Figure 2 This is a comparison image of the table area before and after detection.

[0038] Figure 3 This is a comparison chart showing the difference between preprocessing an irregular table and its preprocessing steps.

[0039] Figure 4 Images showing the before and after correction of a curved boundary table.

[0040] Figure 5 This is a 10x10 perspective-transformed table image.

[0041] Figure 6 This is the result image after recognizing a bordered table. Detailed Implementation

[0042] To describe the present invention in more detail, the technical solution of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0043] like Figure 1 As shown, the table structure detection method for flattening irregular table images according to the present invention includes the following steps:

[0044] Step 1: Image table region detection.

[0045] By feeding the original image tables into a deep learning object detection model, the model can detect the locations of each table region in the entire image and extract the tables. This algorithm can then produce a cropped table detection region, such as... Figure 2 As shown.

[0046] Step 2: Check the form for compliance.

[0047] For the existing cropped table, a standardization check is required. Adaptive thresholding is used to binarize and dilate the grayscale image. Edge contour detection is performed to obtain the coordinates of each group within the table's contours. Area processing is used to find the coordinates of the outermost table contour. Then, a function fitting is performed on each side length to calculate the coordinates of the four corners. Finally, it is determined whether the fitted graph of the corner coordinates is a matrix. The specific process is as follows:

[0048] 2.1 Convert the cropped table image to a grayscale image, and then use an adaptive threshold to binarize and dilate the grayscale image;

[0049] 2.2 Then, the image is subjected to contour edge detection so that the border lines of the entire table can be detected, and the detected contour lines are expanded again.

[0050] 2.3 Calculate the contour coordinates of each group from the detected contour lines, and find the group with the largest contour coordinates by the contour area, and set this group of coordinates as the contour of the outer area of ​​the table.

[0051] 2.4 In this set of punctuation values, find the coordinates of the top left, bottom left, top right and bottom right corners of the table outline, calculate whether the figure fitted by the corner coordinates is a matrix, and thus determine whether the table is standard. When the coordinates of the target outline satisfy the following formula, it is considered a standard table.

[0052] y1∈[y2-5,y2+5]&&y3∈[y4-5,y4+5]

[0053] x1∈[x3-10,x3+10]&&x2∈[x4-10,x4+10]

[0054] In the formula: the coordinates of the top left corner are (x1, y1), the coordinates of the top right corner are (x2, y2), the coordinates of the bottom left corner are (x3, y3), and the coordinates of the bottom right corner are (x4, y4).

[0055] Step 3: Preprocessing of the regional table.

[0056] For the already judged tables, preprocessing of the table image is performed for non-standard tables. For the four curve functions fitted in the previous step, the lengths of the four curves are calculated and normalized. 10*10 key points are selected on each side, and perspective transformation is used to flatten the curve functions, such as... Figure 3 and Figure 4 As shown, the specific process is as follows:

[0057] 3.1 Convert the irregular table image to a grayscale image, and then use an adaptive threshold to perform binarization and dilation operations on the grayscale image;

[0058] 3.2 Then, the image is subjected to contour edge detection so that the border lines of the entire table can be detected, and the detected contour lines are expanded again.

[0059] 3.3 Calculate the contour coordinates of each group from the detected contour lines, and find the group with the largest contour coordinates by the contour area, and set this group of coordinates as the contour of the outer area of ​​the table.

[0060] 3.4 In this set of punctuation values, find the coordinates of the top left, bottom left, top right, and bottom right corners of the table outline, and take the inner boundary points of the four points. In this way, you can find the punctuation coordinates of the four sides and store them into four lists representing each side in turn.

[0061] 3.5 Next, the coordinate values ​​of the contour line points are fitted into four curve functions, and the lengths of the four curves are calculated and normalized. The method for fitting the curves is linear least squares, and the basic idea is:

[0062] f(x) = a1r1(x) + a2r2(x) + ... + a m r m (x)

[0063] In the formula: r k (x) represents a selected set of linearly independent functions, a k The coefficients are undetermined (k = 1, 2, 3, ..., m).

[0064] When fitting curves for table borders, the highest order term is typically 3. The formula for calculating the length of the fitted border curve is as follows:

[0065]

[0066] SUM = I1 + I2 + ... + I m-1

[0067] In the formula: (x i ,y i ) are the coordinates of the fitted curve, I i is the length between two adjacent coordinate points, and m is the width of the entire curve.

[0068] 3.6 Select 10*10 key points on each function and flatten the curve function using perspective transformation. The formula for perspective transformation is:

[0069]

[0070] In the formula: (u,v) represents the original image coordinates, (x,y) is the image coordinates after perspective transformation, x=x′ / w′, y=y′ / w′, and w is the normalization parameter. The transformation matrix is ​​in 3×3 form, which visually projects the image onto a new viewing plane. After perspective transformation, it looks like... Figure 5 As shown.

[0071] Step 4: Recognize bordered tables.

[0072] After preprocessing the entire table, bordered table structure detection is performed using image processing techniques. First, the horizontal and vertical erosion and dilation kernels are determined using different table images. After the erosion operation, the corresponding dilation operation is performed, resulting in an image with horizontal lines and a vertical image. Finally, these images are combined to obtain a raster cell image. Then, the bounding box of each cell is extracted from the grid cell image, and a text file containing the information of each cell is generated, such as... Figure 6 As shown, the specific process is as follows:

[0073] 4.1 The preprocessed table is first processed into grayscale, then the grayscale image is binarized using an adaptive threshold, and then image preprocessing is performed.

[0074] 4.2 Determine the kernels for horizontal and vertical erosion and dilation, and then perform erosion and dilation processing on the processed table image to obtain an image with horizontal lines and an image with vertical lines. The formulas used for image dilation processing are as follows:

[0075] dilation(x,y)=max (x′,y′):k(x′,y′)≠0 I(x+x′,y+y′)

[0076] Where: I is the image to be operated on, k is the convolution kernel, x' and y' are the sizes of the convolution kernel, (x,y) are the pixel coordinates of the image, and the value of the pixel after convolution is equal to the maximum value in the range x'*y' centered on the pixel.

[0077] The formula for image erosion processing is:

[0078] erosion(x,y)=min (x′,y′):k(x′,y′)≠0 i(x+x′,y+y′)

[0079] Furthermore, since the image has been binarized, there are only two possibilities regarding the maximum and minimum values ​​around the kernel, allowing for either erosion or dilation operations.

[0080] 4.3 The two images are combined to obtain a complete raster unit image. After further erosion processing, the obtained image is preprocessed and the coordinate data of each cell is obtained through edge detection.

[0081] The above description of the embodiments is provided to enable those skilled in the art to understand and apply the present invention. Those skilled in the art can readily make various modifications to the above embodiments and apply the general principles described herein to other embodiments without creative effort. Therefore, the present invention is not limited to the above embodiments, and any improvements and modifications made to the present invention by those skilled in the art based on the disclosure thereof should be within the scope of protection of the present invention.

Claims

1. A method for detecting table structure in irregular table images by flattening them, comprising the following steps: (1) Perform table region detection on the original image to obtain table region images; (2) Perform a table standardization check on the table area image. The specific implementation process is as follows: 2.1 Convert the table area image to grayscale, and then use adaptive thresholding to perform binarization and dilation operations on the grayscale image; 2.2 Perform contour edge detection on the image processed in step 2.1, detect all the table borders in the image, and expand these borders. 2.3 Based on the coordinate position information of each frame line, find the outline with the largest area in the image. This outline is the table outline. 2.4 Determine if the table is standardized based on the coordinate information of the four vertices of the table outline; (3) Preprocess the images of table areas that do not conform to the specifications; (4) Recognize the borders of the table area image.

2. The table structure detection method according to claim 1, characterized in that: The specific implementation of step (1) is as follows: input the original image containing the table into the pre-trained deep learning target detection model to detect the regional location information of the table in the output image, and then extract the table to obtain a cropped table region image.

3. The table structure detection method according to claim 1, characterized in that: The criteria for determining whether a table is compliant in step 2.4 are as follows: if all of the following conditions are met, the table is deemed to be compliant; otherwise, the table is deemed not compliant. y1∈[y2-5,y2+5]&y3∈[y4-5,y4+5] x1∈[x3-10,x3+10]&x2∈[x4-10,x4+10] Wherein: the coordinates of the top left corner vertex of the table outline are (x1, y1), the coordinates of the top right corner vertex are (x2, y2), the coordinates of the bottom left corner vertex are (x3, y3), and the coordinates of the bottom right corner vertex are (x4, y4).

4. The table structure detection method according to claim 1, characterized in that: The specific implementation process of step (3) is as follows: 3.1 Convert the non-compliant table area image to a grayscale image, and then use an adaptive threshold to perform binarization and dilation operations on the grayscale image; 3.2 Perform contour edge detection on the image processed in step 3.1, detect all the table borders in the image, and expand these borders. 3.3 Based on the coordinate position information of each frame line, find the outline with the largest area in the image. This outline is the table outline. 3.4 For the four curves of the table outline, the curve functions of these four curves are fitted by the least squares method using the coordinates of the outline points obtained by edge detection. 3.5 For any curve, subdivide it into multiple segments. Each segment can be regarded as a straight line. Use the curve function to calculate the coordinates of the first and last points of each straight line, and then calculate the length of each straight line. The sum of these segments is the length of the curve. 3.6 Using the coordinate information of the four vertices of the table outline and the lengths of the four curves, the table area image is corrected through perspective transformation.

5. The table structure detection method according to claim 4, characterized in that: The specific implementation of step 3.6 is as follows: First, the table outline is divided into multiple blocks of 10×10. The coordinates of the four vertices of each block are calculated based on the coordinate information of the four vertices and the length of the four curves. Then, the two curves at the bottom and left of the table outline are used as the x-axis and y-axis in the new coordinate system, respectively. The coordinates of the four vertices of each block in the new coordinate system are calculated. Then, based on the coordinates of these points in the original coordinate system and the new coordinate system, the table area image is projected onto a new viewing plane through perspective transformation, thereby completing the correction.

6. The table structure detection method according to claim 1, characterized in that: The specific implementation process of step (4) is as follows: 4.1 Convert the table area image to grayscale, then use adaptive thresholding to binarize the grayscale image, and perform image preprocessing; 4.2 The preprocessed image is subjected to erosion and dilation processes in sequence to obtain an image with horizontal lines and an image with vertical lines; 4.3 Combine the horizontal and vertical line images to obtain a complete raster unit image, and then perform erosion processing on the image. 4.4 Finally, edge detection is used to obtain the coordinate data of each cell in the raster unit image, and a text file containing the information of each cell is generated.

7. The table structure detection method according to claim 6, characterized in that: The formula for dilating the image in step 4.2 is as follows: dilation(x,y)=max (x′,y′):k(x′,y′)≠0 I(x+x′,y+y′) The formula for erosion processing of an image is as follows: erosion(x,y)=min (x′,y′):k(x′,y′)≠0 I(x+x′,y+y′) Where: I is the image to be operated on, k is the convolution kernel, x' and y' are the sizes of the convolution kernel, (x,y) are the pixel coordinates of the image, and the value of the pixel after convolution is equal to the maximum value in the range x'*y' centered on the pixel.

Images