Table image processing method, system, device, medium and program product

Abstract

The present application provides a table image processing method that can be applied to the field of image processing technology. The processing method includes the following steps: obtaining a table image; preprocessing the table image; extracting all straight lines in the table image, and correcting the table image according to the inclination angle of the straight lines; checking the availability of the straight lines; and removing unusable straight lines to obtain a table image with a specified format. According to the table image processing method of the present application, an image processing method is used to detect and analyze the table, which can quickly and effectively extract the table in the image. It is applicable to all styles of tables, and is convenient for later use of table images with a specified format to assist image recognition applications and other operations.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method, system, device, medium and program product for processing table images. Background Art

[0002] Currently, table files account for the majority of daily work. Since various table files have no fixed format, it becomes more complicated to recognize the text in the table. Existing technologies usually use deep learning to learn various tables, but it requires collecting a large amount of table data for training, which invisibly increases a lot of work. Summary of the Invention

[0003] This application aims to solve at least one of the technical problems existing in the prior art.

[0004] For example, the present application provides a method for processing table images, which uses images to directly process tables and obtains paperless data storage by extracting straight lines, which can greatly facilitate the storage and access of various types of data.

[0005] A first aspect of the present application provides a method for processing a table image, comprising the following steps:

[0006] Get the table image;

[0007] Preprocessing the table image;

[0008] extracting all straight lines in the table image, and correcting the table image according to the inclination angles of the straight lines;

[0009] checking the availability of the straight line;

[0010] Unusable straight lines are removed to obtain a table image with a specified format.

[0011] According to the table image processing method of the present application, an image processing method is used to detect and analyze the table, which can quickly and effectively extract the table in the picture. It can be applied to tables of all styles, and is convenient for later use of table images with specified formats to assist image recognition applications and other operations.

[0012] Furthermore, preprocessing the table image includes:

[0013] performing a binarization process on the table image;

[0014] Hole filling is performed on the table image.

[0015] Furthermore, performing binarization processing on the table image includes:

[0016] Obtaining the grayscale value of each pixel in the table image;

[0017] Compare the grayscale value of each pixel with the grayscale threshold to classify each pixel into background pixels and handwriting pixels, wherein pixels with grayscale values ​​less than the grayscale threshold are background pixels, and pixels with grayscale values ​​greater than the grayscale threshold are handwriting pixels.

[0018] Furthermore, binarization of the table image further includes: setting the grayscale value of the background pixel to 0, and setting the grayscale value of the handwriting pixel to 255.

[0019] Furthermore, the hole filling is performed on the table image, comprising:

[0020] Obtaining holes in the table image;

[0021] Capturing the pixel point closest to the hole;

[0022] Check the grayscale value of the pixel point and fill the hole with the grayscale value of the pixel point.

[0023] Furthermore, extracting all straight lines in the table image and correcting the table image according to the inclination angles of the straight lines includes:

[0024] extracting all straight lines in the table image and classifying the straight lines as at least one of horizontal lines or vertical lines;

[0025] Marking horizontal auxiliary lines and vertical auxiliary lines on the table image;

[0026] Calculate the inclination angle α between each horizontal line and the horizontal auxiliary line m ;

[0027] Calculate the inclination angle β between each vertical line and the vertical auxiliary line n ;

[0028] The least squares algorithm is used to calculate the horizontal tilt angle θ of the table image. 水平 ;

[0029] The least squares algorithm is used to calculate the vertical tilt angle θ of the table image. 竖直 ,

[0030] Where m represents the number of horizontal lines, n represents the number of vertical lines, and both m and n are positive integers;

[0031] According to the tilt angle θ 水平 and the tilt angle θ 竖直 The table image is rectified.

[0032] Furthermore, the inclination angle α between each horizontal line and the horizontal auxiliary line is calculated. m , calculate the inclination angle β between each vertical line and the vertical auxiliary line n ,include:

[0033] Arbitrarily obtain the two coordinate values ​​​​(x 横线1 ,y 横线1 ), (x 横线2 ,y 横线2 );

[0034] The tilt angle α is obtained according to the two horizontal line coordinates m ;as well as

[0035] Arbitrarily obtain the two coordinate values ​​​​(x 竖线1 ,y 竖线1 ), (x 竖线2 ,y 竖线2 );

[0036] The tilt angle β is obtained according to the two vertical line coordinates n .

[0037] Furthermore, the tilt angle α m and the tilt angle β n Calculated by the first formula,

[0038] Among them, the first formula is:

[0039]

[0040]

[0041] Furthermore, the least squares algorithm is used to calculate the horizontal tilt angle θ of the table image. 水平 , calculate the tilt angle θ of the table image in the vertical direction using the least squares algorithm 竖直 ,include:

[0042] All tilt angles α m Arrange from smallest to largest;

[0043] Get the tilt angle α m The maximum and minimum values ​​of ;

[0044] According to the tilt angle α m The maximum and minimum values ​​of the horizontal line angle are calculated;

[0045] Take the 0.001 floating point value of the average value of the horizontal line angle as the horizontal tilt angle θ of the table image 水平 ;as well as

[0046] All tilt angles β n Arrange from smallest to largest;

[0047] Get the tilt angle β n The maximum and minimum values ​​of ;

[0048] According to the tilt angle β n The maximum and minimum values ​​of the vertical line angle are calculated;

[0049] Take the 0.001 floating point value of the average value of the vertical line angle as the vertical tilt angle θ of the table image 竖直 .

[0050] Furthermore, according to the tilt angle θ 水平 and the tilt angle θ 竖直 Correcting the table image, including:

[0051] The tilt angle θ 水平 and the tilt angle θ 竖直 Input the rotation matrix function to get the rotation matrix;

[0052] According to the rotation matrix, an affine transformation is performed on the table image to obtain a corrected table image.

[0053] Furthermore, according to the tilt angle θ 水平 and the tilt angle θ 竖直 Before correcting the table image, first determine whether the table image needs to be corrected.

[0054] When the tilt angle θ 水平 and the tilt angle θ 竖直 When at least one of the values ​​is greater than a tilt threshold, the table image needs to be corrected.

[0055] Furthermore, checking the availability of the straight line includes:

[0056] Check each intersection of the horizontal and vertical lines;

[0057] When the number of intersections on a horizontal line is less than 2, the horizontal line is unusable;

[0058] Check each intersection of the vertical and horizontal lines;

[0059] When the number of intersections on a vertical line is less than 2, the vertical line is unusable.

[0060] The second aspect of the present application provides a table processing system, including: an acquisition module, which is used to acquire a table image; a processing module, which is used to pre-process the table image; a correction module, which is used to extract all straight lines in the table image and correct the table image according to the inclination angle of the straight lines; a verification module, which is used to verify the availability of the straight lines; and an output module, which is used to remove unusable straight lines to obtain a table image with a specified format.

[0061] The third aspect of the present application provides an electronic device, comprising: one or more processors; a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors execute the above-mentioned processing method.

[0062] The fourth aspect of the present application further provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, causes the processor to execute the above-mentioned processing method.

[0063] The fifth aspect of the present application further provides a computer program product, comprising a computer program, which implements the above-mentioned processing method when executed by a processor. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] The above contents and other objects, features and advantages of the present application will become more apparent through the following description of the embodiments of the present application with reference to the accompanying drawings, in which:

[0065] Figure 1 A flowchart of a table image processing method according to an embodiment of the present application;

[0066] Figure 2 is a flowchart of the binarization process according to an embodiment of the present application;

[0067] Figure 3 is a flow chart of hole filling according to an embodiment of the present application;

[0068] Figure 4 is a flow chart of tilt angle correction according to an embodiment of the present application;

[0069] Figure 5 is a flow chart of a method for calculating the horizontal line inclination angle according to an embodiment of the present application;

[0070] Figure 6 is a flow chart of a method for calculating the vertical line inclination angle according to an embodiment of the present application;

[0071] Figure 7 This is a flow chart of a method for calculating a horizontal tilt angle according to an embodiment of the present application;

[0072] Figure 8 is a flow chart of a method for calculating a vertical tilt angle according to an embodiment of the present application;

[0073] Figure 9 is a flow chart of tilt correction according to an embodiment of the present application;

[0074] Figure 10 is a schematic diagram of a table image before tilt correction according to an embodiment of the present application;

[0075] Figure 11 is a schematic diagram of a table image after tilt correction according to an embodiment of the present application;

[0076] Figure 12 is a flow chart of a method for checking horizontal line availability according to an embodiment of the present application;

[0077] Figure 13 This is an enlarged image of part of the table according to the embodiment of this application Figure 1 ;

[0078] Figure 14 is a flow chart of a method for checking vertical line availability according to an embodiment of the present application;

[0079] Figure 15 This is an enlarged image of part of the table according to the embodiment of this application Figure 2 ;

[0080] Figure 16 is a structural block diagram of a table image processing system according to an embodiment of the present application;

[0081] Figure 17 It is a block diagram of an electronic device suitable for implementing a table image processing method according to an embodiment of the present application. DETAILED DESCRIPTION

[0082] Hereinafter, embodiments of the present application will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present application. In the detailed description below, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present application. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessarily confusing the concepts of the present application.

[0083] The terms used herein are only for describing specific embodiments and are not intended to limit this application. The terms "comprise," "include," etc. used herein indicate the presence of the features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0084] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0085] When expressions such as "at least one of A, B and C, etc." are used, they should generally be interpreted in accordance with the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0086] Currently, spreadsheets make up the majority of daily work. For example, every company is involved in filling out and compiling financial statements. If the accumulated data from various reports is simply archived manually on paper, it becomes extremely cumbersome for data storage, review, financial auditing, and accounting. Since spreadsheets have no fixed format, identifying text within them becomes complex. Using image recognition to extract spreadsheet data and perform paperless data storage can greatly facilitate the storage and access of various financial report data. Existing technologies typically use deep learning to learn various spreadsheets, but this requires collecting large amounts of spreadsheet data for training, which invisibly increases the workload.

[0087] A method for processing table images is provided in an embodiment of the present application. Traditional image processing technology is used to detect table lines in images, eliminating the tedious work of collecting and training data like deep learning. The table lines in the target image can be extracted quickly and accurately to assist image recognition technology in further analyzing the image.

[0088] Image processing is the process of analyzing images using computers to achieve desired results. Image processing technology boasts excellent reproducibility, high precision, and a wide range of applications, and is currently widely used in research fields such as computer vision and graphics.

[0089] It should be noted that the table image processing method provided in the embodiment of the present application can generally be executed by a server. Accordingly, the table processing system provided in the embodiment of the present application can generally be set in a server. The table image processing method provided in the embodiment of the present application can also be executed by a server or server cluster that is different from the server and can communicate with the terminal device and / or the server. Accordingly, the table processing system provided in the embodiment of the present application can also be set in a server or server cluster that is different from the server and can communicate with the terminal device and / or the server.

[0090] pass Figures 1-15 The method for processing the table image of the application embodiment is described in detail. Figure 1 The flowchart of the table image processing method according to the embodiment of the present application is schematically shown.

[0091] like Figure 1 As shown, the table image processing method of this embodiment includes the following steps:

[0092] In step S100 , a table image is acquired.

[0093] An electronic device can obtain an image with a table, and this table image is the image that needs to be processed. The electronic device can obtain a locally stored image with a table. It can also receive an image with a table sent by another electronic device. Of course, it is also possible to obtain an image with a table through an image acquisition device installed in the device, for example, by using a camera installed in the device. All of these are reasonable and are not specifically limited here.

[0094] In step S200, the table image is pre-processed.

[0095] For the acquired table image, it is necessary to extract effective features from the image, eliminate irrelevant information in the image, restore useful real information, enhance the detectability of relevant information and simplify the data to the maximum extent, so as to improve the reliability of feature extraction, image segmentation, matching and recognition.

[0096] Among them, for image preprocessing, this application adopts methods such as binarization and hole filling, such as Figure 2 and Figure 3 shown.

[0097] The binarization process of the table image includes steps S211 to S213.

[0098] After binarization, the entire table image can present an obvious black and white effect. In the post-processing process, the collective properties of the table image are related to the positions of points with pixel values ​​of 0 or 255, and the multi-level values ​​of pixels are no longer involved, making the processing simpler. In addition, the amount of data processing and compression is small, and the amount of data in the table image is greatly reduced, thereby highlighting the outline of the target.

[0099] In step S211, the grayscale value of each pixel in the table image is obtained.

[0100] In step S212, the grayscale value of each pixel is compared with the grayscale threshold to classify each pixel into background pixels and handwriting pixels, wherein pixels with grayscale values ​​less than the grayscale threshold are background pixels, and pixels with grayscale values ​​greater than the grayscale threshold are handwriting pixels.

[0101] In step S213, the grayscale value of the background pixel is set to 0, and the grayscale value of the handwriting pixel is set to 255.

[0102] In this application, the function in the OpenCV program is used to perform binarization processing on the table image. A grayscale threshold can be manually specified first. Of course, the grayscale threshold is between 0 and 255, and the binarization processing is performed based on this grayscale threshold.

[0103] To distinguish between handwriting pixels and background pixels, we first obtain the grayscale value of each pixel in the table image. Handwriting pixels include the table and text in the table image, and background pixels include the content other than the table and text. The grayscale value of each pixel is compared with the initially set grayscale threshold. Pixels with grayscale values ​​less than the grayscale threshold are considered background pixels, while pixels with grayscale values ​​greater than the grayscale threshold are considered handwriting pixels. The grayscale value of background pixels is then set to 0, and the grayscale value of handwriting pixels is set to 255, meaning that background elements are black and handwriting elements are white. This binarization process reduces interference with the table image.

[0104] For example, binarizing a table image and classifying its pixels as either text pixels or background pixels, and assigning grayscale values ​​according to the above rules, yields an intermediate image. As can be seen, the table lines in the table image are all black. After binarizing the table image and classifying its pixels, the intermediate image shows the table lines as white and the rest as black.

[0105] After the image is binarized, the holes in the table image are filled, such as Figure 3 As shown, it includes steps S221 to S223.

[0106] In step S221, the holes in the table image are obtained.

[0107] In step S222, the pixel point closest to the hole is captured.

[0108] In step S223 , the grayscale value of the pixel is checked, and the hole is filled with the grayscale value of the pixel.

[0109] After the image is binarized, there are sometimes some holes in the middle image due to incomplete acquisition of the table image or misclassification when classifying according to the grayscale threshold. In order to optimize the middle image, these holes are filled by the nearest neighbor interpolation method, that is, the pixel closest to the hole is determined, and the grayscale value of this pixel is assimilated to the hole, so that the final table is fuller.

[0110] The table image may be tilted after preprocessing. If the table is tilted, it needs to be corrected first. This application calculates whether the entire table image is tilted by the inclination of each straight line, and corrects the tilted table image.

[0111] In step S300 , all straight lines in the table image are extracted, and the table image is corrected according to the inclination angles of the straight lines.

[0112] Extract all the straight lines in the table, determine the inclination of each straight line, and correct the table image based on the inclination of all the straight lines. The flowchart is as follows: Figure 4 As shown, it specifically includes steps S310 to S370.

[0113] It should be noted that the straight line in this application refers to all lines in the table image. When making judgments, the minimum length of the line is used as a straight line, that is, an intersection point is the starting point of the straight line, and along the direction of the straight line, another intersection point adjacent to this intersection point is the end point of the straight line.

[0114] In step S310 , all straight lines in the table image are extracted and classified as at least one of horizontal lines and vertical lines.

[0115] All straight lines in the table image are divided into horizontal lines and vertical lines, and processed separately to obtain a horizontal line set image and a vertical line set image.

[0116] In step S320, horizontal auxiliary lines and vertical auxiliary lines are marked on the table image.

[0117] For the horizontal line set, mark horizontal auxiliary lines on the table image, use the horizontal auxiliary lines as the standard horizontal lines, and compare each horizontal line with the horizontal auxiliary lines. For the vertical line set, mark vertical auxiliary lines on the table image, use the vertical auxiliary lines as the standard vertical lines, and compare each vertical line with the vertical auxiliary lines.

[0118] In step S330, the inclination angle α between each horizontal line and the horizontal auxiliary line is calculated. m .

[0119] Calculate the inclination angle α of each horizontal line to the horizontal auxiliary line according to the horizontal auxiliary line m The specific process is as follows:

[0120] In step S331, two coordinate values ​​(x 横线1 ,y 横线1 ), (x 横线2 ,y 横线2 ).

[0121] In step S332, the tilt angle α is obtained according to the two horizontal line coordinates. m.

[0122] like Figure 5 As shown, randomly select two coordinate points on the horizontal line, and the angle between the horizontal line and the horizontal auxiliary line is the inclination angle α of the horizontal line relative to the horizontal auxiliary line. m , tilt angle α m Obtained through the first formula, the inclination angle of this horizontal line can be calculated by substituting the two coordinate points into the following first formula:

[0123]

[0124] In step S340, the inclination angle β between each vertical line and the vertical auxiliary line is calculated. n .

[0125] Calculate the inclination angle β between each vertical line and the vertical auxiliary line according to the vertical auxiliary line n The specific process is as follows:

[0126] In step S341, two coordinate values ​​(x 竖线1 ,y 竖线1 ), (x 竖线2 ,y 竖线2 );

[0127] In step S342, the tilt angle β is obtained according to the two vertical line coordinates. n .

[0128] The calculation method is the same as that of horizontal lines, such as Figure 6 As shown, take two coordinate points at random on the vertical line, and the angle between the vertical line and the vertical auxiliary line is the inclination angle β of the vertical line relative to the vertical auxiliary line. n , tilt angle β n Obtained through the first formula, the inclination angle of this vertical line can be calculated by substituting the two coordinate points into the following first formula:

[0129]

[0130] It should be noted that step S330 and step S340 are performed in no particular order.

[0131] In step S350, the least squares method is used to calculate the horizontal tilt angle θ of the table image. 水平 , where m represents the number of horizontal lines and n represents the number of vertical lines, and both m and n are positive integers.

[0132] After obtaining the tilt angle of each horizontal line in step S332, the least squares algorithm is used to calculate the tilt angle θ of the table image in the horizontal direction. 水平 ,like Figure 7 The specific steps are as follows:

[0133] In step S351, all the tilt angles α m Arrange from smallest to largest.

[0134] In step S352, the tilt angle α is obtained. m The maximum and minimum values ​​of .

[0135] In step S353, according to the tilt angle α m The maximum and minimum values ​​of the horizontal line angle are calculated.

[0136] In step S354, the average value of the horizontal line angle is taken as the horizontal tilt angle θ of the table image. 水平 .

[0137] In step S360, the least squares algorithm is used to calculate the vertical tilt angle θ of the table image. 竖直 , where m represents the number of horizontal lines and n represents the number of vertical lines, and both m and n are positive integers.

[0138] After obtaining the tilt angle of each vertical line in step S342, the least squares algorithm is used to calculate the tilt angle θ of the table image in the vertical direction. 竖直 ,like Figure 8 The specific steps are as follows:

[0139] In step S361, all the tilt angles β n Arrange from smallest to largest.

[0140] In step S362, the tilt angle β is obtained. n The maximum and minimum values ​​of .

[0141] In step S363, according to the tilt angle β n The maximum and minimum values ​​of the vertical line angle are calculated.

[0142] In step S364, the average value of the vertical line angle is taken as the vertical tilt angle θ of the table image. 竖直 .

[0143] In step S370, according to the tilt angle θ 水平 and the tilt angle θ 竖直 Rectify table images.

[0144] This application uses OpenCV to perform skew correction on table images, employing an affine transformation. Affine transformation is a linear transformation between two-dimensional coordinates, often involving a linear combination of rotations, scaling, and translations, to maintain the flatness of the two-dimensional image. This technique has broad applications in research fields such as image correction and image deformation.

[0145] like Figure 9 The specific steps are as follows:

[0146] In step S371, the tilt angle θ 水平 and the tilt angle θ 竖直 Input the rotation matrix function to get the rotation matrix.

[0147] In step S372, an affine transformation is performed on the table image according to the rotation matrix to obtain a corrected table image.

[0148] In one embodiment, Figure 10 As shown, the table is obviously tilted. First, in order to facilitate calculation, two auxiliary lines, horizontal and vertical, are marked on the table image. Secondly, the least squares algorithm is used to calculate the tilt angle of each straight line. The tilt angles of all vertical and horizontal lines are calculated according to the first formula, and the horizontal line tilt angles are stored in the horizontal line tilt angle dynamic array, and the vertical line tilt angles are stored in the vertical line tilt angle dynamic array pair array. The tilt angles are sorted in ascending order, and finally the horizontal and vertical tilt angles of the table image are obtained. At this time, the rotation matrix function of OpenCV is used to calculate the rotation matrix, and it is applied to the image to perform affine transformation to achieve the tilt correction processing of the table. The final output table image is as follows Figure 11 As shown, the table image is corrected.

[0149] Check all available straight lines in the table image, retain available straight lines, and remove unavailable straight lines.

[0150] In step S400, the availability of straight lines is checked. The horizontal and vertical lines are checked separately as follows:

[0151] When checking the horizontal line, Figure 12 As shown, in step S411, the intersection of each horizontal line and vertical line is checked.

[0152] In step S412, when the number of intersections on the horizontal line is less than 2, the horizontal line is unavailable.

[0153] like Figure 13 As shown, it is an enlarged view of part of the table image. There are three horizontal lines from top to bottom. Check the intersection points of each horizontal line with all the vertical lines. The number of intersection points of the first horizontal line and the third horizontal line with the vertical lines is 2. It can be determined that these two horizontal lines belong to the component lines of the table lines in the table image. The first horizontal line and the third horizontal line will be retained. The number of intersection points of the second horizontal line with the vertical lines is 1. The number of intersection points is less than 2. It can be determined that this horizontal line does not belong to the component lines of the table lines in the table image, and the second horizontal line is unusable.

[0154] When checking the vertical lines, Figure 14 As shown, in step S421, the intersection of each vertical line and horizontal line is checked.

[0155] In step S422, when the number of intersections on the vertical line is less than 2, the vertical line is unavailable.

[0156] like Figure 15 As shown, it is an enlarged image of part of the table image. There are three vertical lines from left to right. Check the intersection points of each vertical line with all the horizontal lines. The number of intersection points of the first vertical line and the third vertical line with the horizontal lines is 2. It can be determined that these two vertical lines belong to the component lines of the table lines in the table image. The first vertical line and the third vertical line will be retained. The number of intersection points of the second vertical line with the horizontal lines is 1. The number of intersection points is less than 2. It can be determined that this vertical line does not belong to the component lines of the table lines in the table image, and the second vertical line is unusable.

[0157] It should be noted that since a straight line is defined as the minimum length of a line, that is, an intersection is the starting point of the straight line, and another intersection adjacent to this intersection along the direction of the straight line is the ending point of the straight line, so if a straight line is usable, there will be at most two and only two intersections. When the straight line has less than two intersections, it can be judged as unusable.

[0158] In step S500 , unusable straight lines are removed to obtain a table image having a prescribed format.

[0159] The unusable lines detected in step S400 are directly removed to obtain a table image with a specified format. The specified format means that after the processing of steps S200 to S400, the table in the table image has become a standard table, which can be further used to assist in other operations such as image recognition applications.

[0160] This application uses an image processing method to perform table detection and analysis on images with tables. Compared with the table detection method based on deep learning, the processing method of this application is simpler and avoids the tedious work of collecting and training a large amount of table data. In addition, compared with the deep learning model, the processing method of this application is less dependent on hardware computing power.

[0161] According to one embodiment of the present application, according to the tilt angle θ 水平 and the tilt angle θ 竖直 Before correcting a table image, you can first determine whether the table image needs correction.

[0162] When the table image is obtained in step S100, it may be tilted or not. When it is tilted, the table image is corrected through step S300. When it is not tilted, the tilt correction part can be skipped directly. After extracting all the straight lines in the table image, the availability of all the straight lines is directly checked.

[0163] There are three cases of inclination of a table image: the inclination may exist only in the horizontal direction, the inclination may exist only in the vertical direction, or the inclination may exist in both the horizontal and vertical directions.

[0164] Whether it is tilted depends on the tilt angle θ 水平 and the tilt angle θ 竖直 Make a judgment, when the tilt angle θ 水平 and the tilt angle θ 竖直 When at least one of the values ​​is greater than the tilt threshold, the table image needs to be corrected.

[0165] Based on experience, the tilt threshold can be 0.02 float. Using 0.02 float (0.02f) as a reference value, after calculating the tilt angle of the table line, if it is greater than 0.02f, it means that the table in the table image has a certain degree of tilt and needs to be corrected.

[0166] In one embodiment, at an inclination angle θ 水平 Greater than 0.02f and tilt angle θ 竖直 When it is less than or equal to 0.02f, it means that the table in the table image is tilted only in the horizontal direction.

[0167] In another embodiment, at an inclination angle θ 竖直 Greater than 0.02f and tilt angle θ 水平 When it is less than or equal to 0.02f, it means that the table in the table image is tilted only in the vertical direction.

[0168] In yet another embodiment, at an inclination angle θ 竖直 and the tilt angle θ 水平 When both are greater than 0.02f, it means that the table in the table image is tilted only in the vertical and horizontal directions.

[0169] It can be understood that 0.02f represents 0.02 floating point type. Floating point type can be understood as single-precision float. Both are data types. Simply put, they represent data with decimals, and the decimal point can float in different positions of the corresponding binary.

[0170] According to the table image processing method of the present application, an image processing method is used to detect and analyze the table, which can quickly and effectively extract the table in the picture. It can be applied to tables of all styles, and is convenient for later use of table images with specified formats to assist image recognition applications and other operations.

[0171] Based on the above table image processing method, this application also provides a table processing system. Figure 16 The device is described in detail.

[0172] Figure 16 The structural block diagram of the table processing system according to an embodiment of the present application is schematically shown.

[0173] like Figure 16 As shown, the table processing system 600 of this embodiment includes: an acquisition module 610 , a processing module 620 , a correction module 630 , a verification module 640 and an output module 650 .

[0174] The acquisition module 610 is used to acquire the table image. In one embodiment, the acquisition module 610 can be used to perform the operation S100 described above, which will not be described in detail here.

[0175] The processing module 620 is used to pre-process the table image. In one embodiment, the processing module 620 can be used to perform the operation S200 described above, which will not be repeated here.

[0176] The correction module 630 is used to extract all straight lines in the table image and correct the table image according to the inclination angles of the straight lines. In one embodiment, the correction module 630 can be used to perform the operation S300 described above, which will not be repeated here.

[0177] The checking module 640 is used to check the availability of the straight line. In one embodiment, the checking module 640 can be used to perform the operation S400 described above, which will not be described in detail here.

[0178] The output module 650 is used to remove unusable straight lines to obtain a table image with a specified format. In one embodiment, the output module 650 can be used to perform the operation S500 described above, which will not be repeated here.

[0179] According to the table image processing method of the present application, an image processing method is used to detect and analyze the table, which can quickly and effectively extract the table in the picture. It can be applied to tables of all styles, and is convenient for later use of table images with specified formats to assist image recognition applications and other operations.

[0180] According to an embodiment of the present application, any multiple modules among the acquisition module 610, the processing module 620, the correction module 630, the inspection module 640, and the output module 650 can be combined into one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present application, at least one of the acquisition module 610, the processing module 620, the correction module 630, the inspection module 640, and the output module 650 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or can be implemented in any one of the three implementation methods of software, hardware, and firmware, or in an appropriate combination of any of them. Alternatively, at least one of the acquisition module 610 , the processing module 620 , the correction module 630 , the inspection module 640 , and the output module 650 may be at least partially implemented as a computer program module, which may perform corresponding functions when executed.

[0181] Figure 17 A block diagram of an electronic device suitable for implementing a table image processing method according to an embodiment of the present application is schematically shown.

[0182] like Figure 17 As shown, the electronic device 700 according to an embodiment of the present application includes a processor 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage portion 708 into a random access memory (RAM) 703. The processor 701 may, for example, include a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a dedicated microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 701 may also include an onboard memory for caching purposes. The processor 701 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present application.

[0183] Various programs and data required for the operation of the electronic device 700 are stored in the RAM 703. The processor 701, ROM 702, and RAM 703 are connected to each other via a bus 704. The processor 701 performs various operations of the method flow according to the embodiment of the present application by executing the programs in the ROM 702 and / or RAM 703. It should be noted that the programs may also be stored in one or more memories other than the ROM 702 and the RAM 703. The processor 701 may also perform various operations of the method flow according to the embodiment of the present application by executing the programs stored in the one or more memories.

[0184] According to an embodiment of the present application, the electronic device 700 may further include an input / output (I / O) interface 705, which is also connected to the bus 704. The electronic device 700 may further include one or more of the following components connected to the I / O interface 705: an input portion 706 including a keyboard, a mouse, etc.; an output portion 707 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and a speaker; a storage portion 708 including a hard disk; and a communication portion 709 including a network interface card such as a LAN card or a modem. The communication portion 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. A removable medium 711, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed in the drive 710 as needed, so that a computer program read therefrom can be installed into the storage portion 708 as needed.

[0185] This application also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments, or may exist independently and not be incorporated into the device / apparatus / system. The computer-readable storage medium carries one or more programs, and when the one or more programs are executed, the method according to the embodiments of this application is implemented.

[0186] According to an embodiment of the present application, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, it may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, apparatus, or device. For example, according to an embodiment of the present application, the computer-readable storage medium may include the ROM 702 and / or RAM 703 described above and / or one or more memories other than ROM 702 and RAM 703.

[0187] The present application also includes a computer program product, which includes a computer program containing program code for executing the method shown in the flowchart. When the computer program product is executed in a computer system, the program code is used to enable the computer system to implement the item recommendation method provided in the present application.

[0188] The computer program executes the above functions defined in the system / device of the embodiment of the present application when the computer program is executed by the processor 701. According to the embodiment of the present application, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0189] In one embodiment, the computer program may be stored on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may be transmitted and distributed in the form of a signal on a network medium, downloaded and installed via the communication portion 709, and / or installed from a removable medium 711. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to wireless, wired, or any suitable combination thereof.

[0190] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 709, and / or installed from a removable medium 711. When the computer program is executed by the processor 701, the above-described functions defined in the system of the embodiment of the present application are performed. According to the embodiment of the present application, the systems, devices, means, modules, units, etc. described above can be implemented by computer program modules.

[0191] According to an embodiment of the present application, the program code for executing the computer program provided in the embodiment of the present application can be written in any combination of one or more programming languages, specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect through the Internet).

[0192] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the above-mentioned module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of the boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0193] Those skilled in the art will appreciate that various combinations and / or combinations of features described in the various embodiments and / or claims of this application may be made, even if such combinations or combinations are not explicitly described in this application. In particular, various combinations and / or combinations of features described in the various embodiments and / or claims of this application may be made, without departing from the spirit and teachings of this application. All such combinations and / or combinations fall within the scope of this application.

[0194] Throughout this specification, reference to terms such as "one embodiment," "some embodiments," "examples," "specific examples," or "some examples" means that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0195] The embodiments of the present application have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of the present application. Although each embodiment has been described separately above, this does not mean that the measures in each embodiment cannot be advantageously used in combination. The scope of the present application is defined by the appended claims and their equivalents. Without departing from the scope of the present application, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present application.

Claims

1. A method for processing a table image, characterized in that: The following steps are involved: Get the table image; Preprocessing the table image; Extract all the straight lines in the table image, and correct the table image according to the inclination angles of the straight lines, wherein the straight lines are all the lines in the table image, and the lines are of minimum length, wherein the extracting all the straight lines in the table image and correcting the table image according to the inclination angles of the straight lines include: extracting all the straight lines in the table image, and classifying the straight lines as at least one of horizontal lines or vertical lines; marking horizontal auxiliary lines and vertical auxiliary lines on the table image; and calculating the inclination angles of each horizontal line and the horizontal auxiliary line. ; Calculate the inclination angle between each vertical line and the vertical auxiliary line ; Use the least squares algorithm to calculate the tilt angle of the table image in the horizontal direction ; Use the least squares algorithm to calculate the tilt angle of the table image in the vertical direction , where m represents the number of horizontal lines and n represents the number of vertical lines, both m and n are positive integers; according to the inclination angle and tilt angle Correcting the table image; wherein the minimum length is based on the length between the intersection point as the starting point and the intersection point as the ending point in the straight line; Checking the availability of the straight line; wherein checking the availability of the straight line includes: checking the intersection of each horizontal line and the vertical line; when the number of intersections on the horizontal line is less than 2, the horizontal line is unavailable; checking the intersection of each vertical line and the horizontal line; when the number of intersections on the vertical line is less than 2, the vertical line is unavailable; Unusable straight lines are removed to obtain a table image with a prescribed format.

2. The processing method according to claim 1, characterized in that Preprocessing the table image includes: performing a binarization process on the table image; Hole filling is performed on the table image.

3. The processing method according to claim 2, characterized in that The binarization processing of the table image includes: Obtaining the grayscale value of each pixel in the table image; Compare the grayscale value of each pixel with the grayscale threshold to classify each pixel into background pixels and handwriting pixels, wherein pixels with grayscale values ​​less than the grayscale threshold are background pixels, and pixels with grayscale values ​​greater than the grayscale threshold are handwriting pixels.

4. The processing method according to claim 3, characterized in that The binarization processing of the table image also includes setting the grayscale value of the background pixel to 0 and setting the grayscale value of the handwriting pixel to 255.

5. The processing method according to claim 2, characterized in that: Filling holes in the table image, including: Obtaining holes in the table image; Capturing the pixel point closest to the hole; Check the grayscale value of the pixel point and fill the hole with the grayscale value of the pixel point.

6. The processing method according to claim 1, characterized in that Calculate the inclination angle of each horizontal line and the horizontal auxiliary line , calculate the inclination angle between each vertical line and the vertical auxiliary line ,include: Arbitrarily obtain the two coordinate values ​​​​on each horizontal line ( ), ( ); Obtain the tilt angle based on the two horizontal line coordinates ;as well as Arbitrarily obtain the two coordinate values ​​​​of each vertical line ( ), ( ); Obtain the tilt angle based on the two vertical line coordinates .

7. The processing method according to claim 6, characterized in that The tilt angle and the tilt angle Calculated by the first formula, Among them, the first formula is: ； 。 8. The processing method according to claim 1, characterized in that The least squares algorithm is used to calculate the horizontal tilt angle of the table image. , using the least squares algorithm to calculate the vertical tilt angle of the table image ,include: All tilt angles Arrange from smallest to largest; Get the tilt angle The maximum and minimum values ​​of ; According to the tilt angle The maximum and minimum values ​​of the horizontal line angle are calculated; Take the 0.001 floating point value of the average value of the horizontal line angle as the horizontal tilt angle of the table image ;as well as All tilt angles Arrange from smallest to largest; Get the tilt angle The maximum and minimum values ​​of ; According to the tilt angle The maximum and minimum values ​​of the vertical line angle are calculated; Take the 0.001 floating point value of the average value of the vertical line angle as the vertical tilt angle of the table image .

9. The processing method according to claim 1, characterized in that: According to the tilt angle and tilt angle Correcting the table image, including: The tilt angle and tilt angle Input the rotation matrix function to get the rotation matrix; According to the rotation matrix, an affine transformation is performed on the table image to obtain a corrected table image.

10. The processing method according to claim 8, characterized in that: According to the tilt angle and tilt angle Before correcting the table image, first determine whether the table image needs to be corrected. When the tilt angle and tilt angle When at least one of the values ​​is greater than a tilt threshold, the table image needs to be corrected.

11. A form processing system comprising: An acquisition module, configured to acquire a table image; A processing module, configured to pre-process the table image; A correction module, the correction module is used to: extract all straight lines in the table image, and correct the table image according to the inclination angle of the straight lines, wherein the straight lines are all lines in the table image, and the lines are of minimum length, wherein the extraction of all straight lines in the table image and the correction of the table image according to the inclination angle of the straight lines include: extracting all straight lines in the table image and classifying the straight lines as at least one of horizontal lines or vertical lines; marking horizontal auxiliary lines and vertical auxiliary lines on the table image; calculating the inclination angle of each horizontal line with the horizontal auxiliary line ; Calculate the inclination angle between each vertical line and the vertical auxiliary line ; Use the least squares algorithm to calculate the tilt angle of the table image in the horizontal direction ; Use the least squares algorithm to calculate the tilt angle of the table image in the vertical direction , where m represents the number of horizontal lines and n represents the number of vertical lines, both m and n are positive integers; according to the inclination angle and tilt angle Correcting the table image; wherein the minimum length is based on the length between the intersection point as the starting point and the intersection point as the ending point in the straight line; A verification module, the verification module is used to verify the availability of the straight line; wherein verifying the availability of the straight line includes: checking the intersection of each horizontal line and the vertical line; when the number of intersections on the horizontal line is less than 2, the horizontal line is unavailable; checking the intersection of each vertical line and the horizontal line; when the number of intersections on the vertical line is less than 2, the vertical line is unavailable; and An output module is used to remove unusable straight lines to obtain a table image with a specified format.

12. An electronic device comprising: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors are enabled to execute the table processing method according to any one of claims 1 to 10. 13 . A computer-readable storage medium having executable instructions stored thereon, wherein when the instructions are executed by a processor, the processor executes the table processing method according to claim 1 . 14 . A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the table processing method according to claim 1 is implemented.

Images