Method and device for identifying table in image, computer equipment and readable storage medium

Abstract

The invention provides a method and a device for identifying a table in an image, computer equipment and a readable storage medium. The method for identifying the table in the image comprises the following steps: detecting cells in a to-be-processed image to obtain a plurality of cell coordinates; extracting a cell image from the to-be-processed image by using the cell coordinates; identifying a text in the cell image; constructing a table in the to-be-processed image by using the plurality of cell coordinates; and correspondingly outputting the text in each cell image into the table. According to the invention, the accuracy of table identification in the image can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image recognition processing, and in particular to a method, device, computer equipment and readable storage medium for recognizing a table in an image. Background Art

[0002] The quota table, also known as the cost quota table, is a table that quantitatively determines the materials, labor, machinery and other resources required in the construction process according to the characteristics of the project and the construction method in the construction industry, and calculates the budget data of various construction projects through unit prices. It provides an important basis for the budget and management of construction projects. It is generally compiled and issued by the provincial quota station, and the format varies from region to region.

[0003] With the prevalence of electronic data, many scenarios require the recognition of images corresponding to this type of table to obtain accurate content information. The prior art uses a multi-model based on deep learning, specifically using a layout analysis model + table recognition model + text detection model + text recognition model for joint recognition. However, the inventors have found that the accuracy of table recognition based on the deep learning model is very low on the quota table. The main reason is that the training sample table styles of the open source table recognition model are very different. Specifically, the training samples are based on public table data, and merged cells mainly exist in the first row and the first column. The quota table information expression is richer, and merged cells will appear in more positions, especially the sub-item part is more varied. In order to improve the recognition rate of the quota table by the table recognition model, it is necessary to synthesize massive quota table sample data and invest huge hardware resources. In addition, in the combination of multiple models, each model will have a certain error rate, and the error rate will increase after superposition.

[0004] Therefore, how to improve the recognition accuracy of quota tables and other types of tables in images has become a technical problem that needs to be urgently solved in this field. Summary of the invention

[0005] The purpose of the present invention is to provide a method, device, computer equipment and readable storage medium for recognizing a table in an image, so as to solve the technical problems in the prior art.

[0006] On the one hand, to achieve the above objective, the present invention provides a method for recognizing a table in an image.

[0007] The method for recognizing a table in an image includes: detecting cells in an image to be processed to obtain a plurality of cell coordinates; extracting a cell image from the image to be processed using the cell coordinates; recognizing text in the cell image; constructing a table in the image to be processed using the plurality of cell coordinates; and outputting the text in each cell image to the table.

[0008] Furthermore, the method also includes: before the step of detecting cells in the image to be processed, performing overall perspective correction on the image to be processed; after the step of extracting the cell image in the image to be processed using the cell coordinates, identifying text outside the cell image; after the step of constructing a table in the image to be processed using multiple cell coordinates, outputting the text outside the cell image outside the table.

[0009] Furthermore, the step of performing overall perspective correction on the image to be processed includes: performing grayscale processing and binarization processing on the image to be processed in sequence to obtain a first image; identifying the quadrilateral with the largest area in the first image; extending the four sides of the quadrilateral with the largest area to the boundary of the first image; translating the four extended sides outward so that the extension line of each side can pass through at least one corner of the first image; calculating the four intersection points of the four sides after translation; and performing perspective correction on the portion of the image to be processed defined by the four intersection points.

[0010] Furthermore, the step of identifying the quadrilateral with the largest area in the first image includes: step S11: setting parameter values ​​of the contour search function and parameter values ​​of the polygon fitting function; step S12: detecting the contour boundary line in the first image through the contour search function, performing polygon fitting on the contour boundary line in the first image through the polygon fitting function, and inputting the quadrilateral with the largest area into the quadrilateral set among the fitted polygons; step S13: judging whether a preset loop end condition is met, if the loop end condition is not met, returning to step S11, if the loop end condition is met, executing step S14; step S14: outputting the quadrilateral with the largest area in the first image according to the quadrilateral set.

[0011] Furthermore, the cells in the image to be processed include a first cell, and the step of detecting the cells in the image to be processed includes: identifying the contour boundary line in the image to be processed; performing polygon fitting on the contour boundary line; filtering out quadrilaterals from the fitted polygons; performing rectangle fitting on the filtered quadrilaterals; and obtaining the first cell coordinates based on the fitted rectangle.

[0012] Furthermore, the step of filtering out quadrilaterals from the fitted polygons includes: determining the maximum area occupied by a single word according to the image resolution of the image to be processed and the font size in the table; and extracting quadrilaterals whose areas are larger than the maximum area occupied by the single word from the fitted polygons.

[0013] Further, the step of performing rectangular fitting on the filtered quadrilateral includes: taking out the four vertices of the quadrilateral, and sorting them in the order of upper left, upper right, lower right, and lower left according to the positions of the four vertices, and obtaining the coordinates of the four vertices as (X [左上] , Y [左上] ), (X [右上] , Y [右上] ), (X [右下] , Y [右下] ), (X [左下] , Y [左下] ); recalculate the coordinates of the upper left and lower right vertices as follows: X [左上] ' = (X [左上] + X [左下] ) / 2, Y [左上] ' = (Y [左上] + X [右上] ) / 2,X [右下] ' = (X [右下] + X [右上] ) / 2, Y [右下] ' = (Y [右下] + Y [左下] ) / 2; recalculate the coordinates of the upper right vertex and the lower left vertex as follows: X [右上] '= X [右下] ', Y [右上] ' = Y [左上] ', X [左下] ' = X [左上] ', Y [左下] '=Y [右下] ', where the four vertices of each quadrilateral (X [左上] ', Y [左上] '), (X [右上] ', Y [右上] '), (X [右下] ', Y [右下] '), (X [左下] ', Y [左下] ') form a rectangle.

[0014] Furthermore, the step of obtaining the first cell coordinates based on the fitted rectangle includes: generating a blank image according to the size of the image to be processed; drawing the fitted rectangle into the blank image to obtain a second image; performing edge detection on the second image and extracting line segments to obtain a line segment set; identifying horizontal and vertical lines in the line segment set to obtain a horizontal line set and a vertical line set; and determining the first cell coordinates based on the intersection points of the horizontal and vertical lines in the horizontal line set and the vertical line set.

[0015] Furthermore, the steps of identifying the horizontal and vertical lines in the line segment set and obtaining the horizontal and vertical line sets include: judging whether the longitudinal coordinates of the two endpoints of the line segment are equal; if the longitudinal coordinates of the two endpoints of the line segment are equal, determining that the line segment is a horizontal line; judging whether the horizontal coordinates of the two endpoints of the line segment are equal; if the horizontal coordinates of the two endpoints of the line segment are equal, determining that the line segment is a vertical line; if the horizontal and longitudinal coordinates of the two endpoints of the line segment are not equal, calculating the slope of the line segment; when the absolute value of the slope of the line segment is less than 1, determining that the line segment is a horizontal line, and when the absolute value of the slope of the line segment is greater than 1, determining that the line segment is a vertical line.

[0016] Further, the step of determining the first cell coordinates according to the intersection points of the horizontal and vertical lines in the horizontal line set and the vertical line set includes: searching the horizontal lines in the horizontal line set for the horizontal lines in the same row to obtain the horizontal line row; traversing the horizontal line row to determine the relationship between adjacent horizontal lines in the row extension direction, wherein, if the current adjacent horizontal lines have overlapping parts or the gaps therein are less than a first preset number of pixels, the adjacent horizontal lines are merged into a new horizontal line, and the judgment is performed with the next adjacent horizontal line, and finally the horizontal line row is processed as a long horizontal line row; searching the vertical lines in the vertical line set for the vertical lines in the same column to obtain the vertical line column; traversing the vertical line column to determine the relationship between adjacent vertical lines in the column extension direction, wherein, if the current adjacent vertical lines have overlapping parts or the gaps therein are less than a second preset number of pixels, the adjacent vertical lines are merged into a new vertical line, and the judgment is performed with the next adjacent vertical line, and finally the vertical line column is processed as a long vertical line column; and determining the first cell coordinates according to the coordinates of the intersection points of the horizontal and vertical lines in the long horizontal line row and the long vertical line column.

[0017] Furthermore, the step of finding horizontal lines in the same row in the horizontal line set to obtain the horizontal line row includes: calculating the ordinate of the midpoint of each horizontal line in the horizontal line set; sorting the horizontal lines in the horizontal line set according to the ordinate of the midpoint; taking the adjacent horizontal lines in the sorted horizontal line set whose ordinates of the midpoints differ by a third preset number of pixels as the horizontal lines in the same row; traversing the horizontal line row to determine the relationship between adjacent horizontal lines in the row extension direction includes: step S21: sorting the horizontal lines in the horizontal line row according to the ordinate of the same end point of the line segment; step S22: determining the relationship between the first horizontal line and the second horizontal line, wherein the initial values ​​of the first horizontal line and the second horizontal line are the The first horizontal line and the second horizontal line in the horizontal line row, when the first horizontal line and the second horizontal line are in a containing relationship, execute step S23; when the first horizontal line and the second horizontal line are in a partially overlapping relationship, or when there is a gap between the first horizontal line and the second horizontal line in the row extension direction, and the gap is less than or equal to the first preset number of pixels, execute step S24; or when there is a gap between the first horizontal line and the second horizontal line in the row extension direction, and the gap is greater than the first preset number of pixels, execute step S26; step S23: determine that the longer horizontal line between the first horizontal line and the second horizontal line is the new horizontal line, and execute step S25; step S2 4: taking the abscissas of the two most distant endpoints on the first horizontal line and the second horizontal line as the abscissas of the new horizontal line, taking the average of the ordinates of the endpoints on the first horizontal line and the second horizontal line as the ordinate of the new horizontal line, and executing step S25; step S25: taking the new horizontal line as the first horizontal line, extracting the next horizontal line in the horizontal line row as the second horizontal line, and returning to step S22, wherein when all the horizontal lines in the horizontal line row are extracted, executing step S27; step S26: storing the first horizontal line in the long horizontal line row, taking the second horizontal line as the first horizontal line, extracting the next horizontal line in the horizontal line row as the second horizontal line, and returning to step S28; step S29: storing the first horizontal line in the long horizontal line row, taking the second horizontal line as the first horizontal line, extracting the next horizontal line in the horizontal line row as the second horizontal line, and returning to step S30; step S40: storing the first horizontal line in the long horizontal line row, taking the second horizontal line as the first horizontal line, extracting the next horizontal line in the horizontal line row as the second horizontal line, and returning to step S31; step S41: storing the first horizontal line in the long horizontal line row, taking the second horizontal line as the first horizontal line, extracting the next horizontal line in the horizontal line row as the second horizontal line, and returning to step S32; step S42: storing the first horizontal line in the long horizontal line row, taking the second horizontal line as the first horizontal line, extracting the next horizontal line in the horizontal line row as the second horizontal line, and returning to step S33; step S43: storing the first horizontal line in the long horizontal line row, taking the second horizontal line as the first horizontal line, extracting the next horizontal line in the horizontal line row as the second horizontal line, and returning to step S34; step S44: storing the first horizontal line in the long horizontal line row, taking the second horizontal line as the first horizontal line, extracting the next horizontal line in the horizontal line row as the second horizontal line, and returning to step S35; Step S22, wherein when all horizontal lines in the horizontal line row are extracted, step S27 is executed; step S27: output the long horizontal line row; the step of finding the vertical lines in the same column in the vertical line set to obtain the vertical line column includes: calculating the horizontal coordinates of the midpoints of each vertical line in the vertical line set; sorting the vertical lines in the vertical line set into columns according to the horizontal coordinates of the midpoints; taking the adjacent vertical lines in the sorted vertical line set whose horizontal coordinates of the midpoints differ by a fourth preset number of pixels as the vertical lines in the same column; traversing the vertical line column, the step of determining the relationship between adjacent vertical lines in the column extension direction includes: step S31: sorting the vertical lines in the vertical line column into columns according to the vertical coordinates of the same end point of the line segment;Step S32: Determine the relationship between the first vertical line and the second vertical line, wherein the initial values ​​of the first vertical line and the second vertical line are the first vertical line and the second vertical line in the vertical line column, when the first vertical line and the second vertical line are in an inclusion relationship, execute step S33, when the first vertical line and the second vertical line are in a partially overlapping relationship, or when there is a gap between the first vertical line and the second vertical line in the column extension direction, and the gap is less than or equal to the second preset pixel number, execute step S34, or when there is a gap between the first vertical line and the second vertical line in the column extension direction, and the gap is greater than the second preset pixel number, execute step S36; Step S33: Determine that the longer vertical line between the first vertical line and the second vertical line is the new vertical line, and execute step S35; Step S34: The ordinates of the two most distant endpoints on the first vertical line and the second vertical line are used as the ordinates of the new vertical line, and the average of the abscissas of the endpoints on the first vertical line and the second vertical line is used as the abscissa of the new vertical line, and step S35 is executed; step S35: the new vertical line is used as the first vertical line, and the next vertical line in the vertical line column is extracted as the second vertical line, and the process returns to step S32, wherein when all the vertical lines in the vertical line column are extracted, step S37 is executed; step S36: the first vertical line is stored in the long vertical line column, the second vertical line is used as the first vertical line, and the next vertical line in the vertical line column is extracted as the second vertical line, and the process returns to step S32, wherein when all the vertical lines in the vertical line column are extracted, step S37 is executed; step S37: the long vertical line column is output. ;

[0018] Furthermore, the step of determining the coordinates of the first cell according to the coordinates of the intersections of the horizontal and vertical lines in the long horizontal line row and the long vertical line column includes: comparing the horizontal and vertical lines in the long horizontal line row and the long vertical line column in pairs to determine whether there is an intersection, obtaining a vertex when an intersection exists, and writing the vertex into a vertex set in sequence according to the row extension direction and the column extension direction; traversing the vertex set, performing the following steps: step S410: obtaining the first vertex, wherein the initial value of the first vertex is the first vertex in the vertex set; step S411: determining whether there is an adjacent next vertex to the first vertex in the row extension direction, if not, executing step S412, and if so, executing Step S413; Step S412: take the next vertex in the vertex set as the first vertex, and return to step S410; Step S413: take the next vertex adjacent to the first vertex in the row extension direction as the second vertex, and take the first vertex as the first vertex of the first cell; Step S414: Determine whether the second vertex has an adjacent next vertex in the column extension direction, if not, execute step S415, if yes, execute step S417; Step S415: Determine whether the second vertex has an adjacent next vertex in the row extension direction, if not, execute step S412, if yes, execute step S416; Step S416: The next vertex adjacent to the second vertex is taken as the second vertex, and the process returns to step S414; step S417: the next vertex adjacent to the second vertex in the column extension direction is taken as the third vertex, and the second vertex is taken as the second vertex of the first cell; step S418: determine whether the third vertex has an adjacent previous vertex in the row extension direction, if not, execute step S419, if yes, execute step S421; step S419: determine whether the third vertex has an adjacent next vertex in the column extension direction, if not, execute step S412, if yes, execute step S420; step S420: the next vertex adjacent to the third vertex in the column extension direction is taken as The third vertex, return to step S418; step S421: take the previous vertex adjacent to the third vertex in the row extension direction as the fourth vertex, and take the third vertex as the third vertex of the first cell; step S422: determine whether there is an adjacent previous vertex for the fourth vertex in the column extension direction, if not, execute step S423, if yes, execute step S425; step S423: determine whether there is an adjacent previous vertex for the fourth vertex in the row extension direction, if not, execute step S412, if yes, execute step S424; step S424: take the previous vertex adjacent to the fourth vertex in the row extension direction as the fourth vertex, and return to step S422;Step S425: take the previous vertex adjacent to the fourth vertex in the column extension direction as the fifth vertex, and take the fourth vertex as the fourth vertex of the first cell; Step S426: determine whether the fifth vertex is the same vertex as the first vertex, if not, execute step S427, if yes, execute step S429; Step S427: determine whether the fifth vertex has an adjacent previous vertex in the column extension direction, if not, execute step S412, if yes, execute step S428; Step S428: take the previous vertex adjacent to the fifth vertex in the column extension direction as the fifth vertex, and return to step S426; Step S429: determine the coordinates of the first cell using the coordinates of the first vertex of the first cell, the second vertex of the first cell, the third vertex of the first cell and the fourth vertex of the first cell. ;

[0019] Further, the image to be processed includes rows that are distinguished alternately by a first background color and a second background color, and the step of identifying the contour boundary lines in the image to be processed includes: converting the foreground text in the image to be processed into black, converting the first background color into black, and converting the second background color into white to obtain a first converted image; converting the foreground text in the image to be processed into black and converting the entire background into white to obtain a second converted image; detecting the contour boundary lines in the first converted image and the second converted image, wherein polygon fitting is performed on the contour boundary lines in the first converted image and the second converted image, respectively, quadrilaterals are filtered out from the fitted polygons, respectively, and rectangle fitting is performed on the filtered quadrilaterals, respectively, to obtain a group of rectangle sets; the step of drawing the fitted rectangles into the blank image includes: drawing both groups of rectangle sets into the blank image.

[0020] Furthermore, the first background color is gray, the second background color is white, the foreground text in the image to be processed is converted to black, the first background color is converted to black, and the second background color is converted to white, and the step of obtaining the first converted image includes: after grayscale processing of the image to be processed, the pixel values ​​of the points in the image to be processed whose pixel values ​​are less than 240 are all set to 0, and the pixel values ​​of the points greater than or equal to 240 are all set to 255; the foreground text in the image to be processed is converted to black, and the entire background is converted to white, and the step of obtaining the second converted image includes: after grayscale processing of the image to be processed, the minimum pixel value of the points in the first background color area is found, and the pixel values ​​of the points in the image to be processed whose pixel values ​​are greater than the minimum pixel value are all set to 255.

[0021] Furthermore, the cells of the image to be processed also include a second cell located within the first cell and without a dividing marker, and the step of identifying the contour boundary line in the image to be processed includes: performing grayscale processing on the image to be processed to obtain a grayscale image; detecting the contour boundary line in the grayscale image; the step of detecting the cells in the image to be processed also includes: identifying a preset character area in the first cell according to text features of preset characters; calculating the center line of each of the preset character areas in the first cell to obtain a center line set corresponding to the first cell; and determining the second cell according to the center line set.

[0022] Further, the step of determining the second cell according to the centerline set includes: step S50: extracting the first edge line, wherein the initial value of the first edge line is the first edge line of the first cell in the row extension direction; step S51: determining the center line adjacent to the first edge line in the centerline set to obtain a reference center line; step S52: judging whether the reference center line is the last center line in the centerline set, if not, executing step S53, if yes, executing step S56; step S53: calculating the distance between the first edge line and the reference center line to obtain a half-line spacing, translating the reference center line in a direction away from the second edge line by the half-line spacing to obtain a second edge line; step S54: determining that the first edge line, the second edge line, and the portion of the first cell intercepted by the first edge line and the second edge line are the second cell; step S55: taking the second edge line as the first edge line, and returning to step S50; step S56: determining that the first edge line, the second edge line of the first cell in the row extension direction, and the portion of the first cell intercepted by the first edge line and the second edge line are the second cell.

[0023] Furthermore, the step of identifying a preset character area in the first cell according to text features of preset characters includes: detecting the text area in the image to be processed to obtain a first text area; cross-judging the first text area and the first cell to determine the first text area in the first cell; and for the first text area in the first cell, filtering out the first text area containing the preset character according to the text features of the preset character to obtain the preset character area in the first cell.

[0024] Furthermore, the step of identifying the text in the cell image includes: detecting a second text area in the cell image; determining a second text area located in the same row among the multiple detected second text areas to obtain a text area row; determining the upper and lower edges of the text area row extending in the row extension direction according to the vertices of the second text area; determining the left and right edges of the text area row according to the boundary line of the cell in the column extension direction; determining the target area using the upper, lower, left and right edges of the text area row; identifying the text in the target area to obtain the text in the cell image.

[0025] Furthermore, the step of identifying the text in the cell image also includes: when the second text area cannot be detected, obtaining preset parameters corresponding to the typeset features of the table; determining the upper and lower borders of the suspected text area in the cell image according to the preset parameters; determining the left and right borders of the suspected text area according to the boundary line of the cell in the column extension direction; and determining the target area using the upper, lower, left and right borders of the suspected text area row.

[0026] On the other hand, to achieve the above objective, the present invention provides a device for recognizing a table in an image.

[0027] The recognition device includes: a detection module, used to detect cells in the image to be processed and obtain multiple cell coordinates; an extraction module, used to extract cell images from the image to be processed using the cell coordinates; a recognition module, used to recognize text in the cell images; and an output module, used to construct a table in the image to be processed using the multiple cell coordinates, and output the text in the cell image to the table.

[0028] To achieve the above objectives, the present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.

[0029] To achieve the above object, the present invention also provides a computer-readable storage medium on which a computer program is stored, and the computer program implements the steps of the above method when executed by a processor.

[0030] The method, device, computer equipment and readable storage medium for recognizing a table in an image provided by the present invention first detect the cells in the image to be processed, then extract the cell image in the image to be processed using the cell coordinates, and recognize the text in the cell image, and finally construct the table in the image to be processed using the cell coordinates, and output the text in the cell image to the table. Through the present invention, the accuracy of table recognition in an image can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings: Figure 1 A flowchart of a method for recognizing a table in an image provided by Embodiment 1 of the present invention; Figure 2 A flow chart of processing a horizontal line row into a long horizontal line row in the method for recognizing a table in an image provided by the first embodiment of the present invention; Figure 3 A flow chart of processing a vertical line column into a long vertical line column in the method for recognizing a table in an image provided by the first embodiment of the present invention; Figure 4 A flow chart of obtaining a first cell through a vertex set in a method for recognizing a table in an image provided by the first embodiment of the present invention; Figure 5 A schematic diagram of perspective correction in the method for recognizing a table in an image provided in the first embodiment of the present invention; Figure 6 A flow chart of identifying the largest quadrilateral in the method for identifying a table in an image provided by the first embodiment of the present invention; Figure 7-Figure 12 This is a recognition effect diagram of table recognition in an image provided by the first embodiment of the present invention; Figures 13 to 16 Another recognition effect diagram of table recognition in an image provided by the first embodiment of the present invention; Fig.17 A flow chart of determining a second cell according to a centerline set in a method for recognizing a table in an image provided by Embodiment 1 of the present invention; Fig.18 and Fig.19 This is another effect diagram of table recognition in an image provided by the first embodiment of the present invention; Fig. 20 A flowchart of a method for recognizing a table in an image provided by Embodiment 2 of the present invention; Fig.21 A block diagram of a device for recognizing a table in an image provided by Embodiment 3 of the present invention; Fig. 22 This is a hardware structure diagram of a computer device provided in Embodiment 4 of the present invention. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical scheme and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0033] Embodiment 1 The embodiment of the present invention provides a method for recognizing a table in an image. By using this method, the recognition accuracy of the table in the image can be improved. Specifically, Figure 1 This is a flow chart of a method for recognizing a table in an image provided by Embodiment 1 of the present invention, such as Figure 1 As shown, the method for recognizing a table in an image provided by this embodiment includes the following steps S101 to S105.

[0034] Step S101: Detect cells in the image to be processed and obtain multiple cell coordinates.

[0035] Optionally, when the image to be processed is an image obtained by photographing a paper form document through a photographing device, before executing step S101, the image to be processed is first perspective corrected to eliminate the perspective deformation in the image caused by the shooting angle, so that the image is closer to the actual shape of the form document, thereby improving the readability and accuracy of the form content, and ensuring more accurate identification of the form for data extraction in subsequent steps.

[0036] It should be noted that if perspective correction is first performed on the image to be processed, the processing of the image to be processed in the subsequent steps is all processing of the image to be processed after perspective correction. For example, in step S101, detecting cells in the image to be processed is detecting cells in the image to be processed after perspective correction; in the following step S102, extracting cell images in the image to be processed using cell coordinates is extracting cell images in the image to be processed after perspective correction using cell coordinates, and so on, which will not be repeated.

[0037] In this step S101, an edge detection algorithm may be used to identify the borders and lines of the table, and further, based on the edge detection, a contour extraction algorithm may be used to identify and extract the structure of the table to obtain the cell coordinates. Alternatively, Hough transform may be used to detect straight lines to find the horizontal and vertical lines of the table, thereby determining the boundaries of the cells and obtaining the cell coordinates. Alternatively, image segmentation technology may be used to separate the table from the background to identify the cell coordinates.

[0038] In one embodiment, a cell in the image to be processed is defined as a first cell. When detecting the cell in the image to be processed in step S101, the specific steps performed include: identifying the contour boundary line in the image to be processed; performing polygon fitting on the contour boundary line; filtering out quadrilaterals from the fitted polygons; performing rectangle fitting on the filtered quadrilaterals; and obtaining the first cell coordinates based on the fitted rectangle.

[0039] Specifically, if the image to be processed is a color image, the original color image is first converted into a grayscale image, and the grayscale image is Gaussian blurred and binarized to obtain a binary image. When identifying the contour boundary lines in the image to be processed, the findContours function of OpenCV is used to find all contours from the above binary image to obtain the contour boundary lines. Among them, the contour in image processing refers to the curve of the boundary of the area with the same grayscale value in the image. The binary image converts the image into an image with only two pixel values ​​(such as black and white). The findContours function of OpenCV is used to extract the boundaries between these black and white areas from the binary image and return the coordinate point set of the contour.

[0040] After obtaining the contour boundary line, polygon fitting is performed on each contour set found through the OpenCV polygon fitting function approxPolyDP, and quadrilaterals are further filtered out from the fitting results to obtain a set of quadrilateral vertices. Further optionally, in one embodiment, when filtering out quadrilaterals from the fitted polygons, the maximum area occupied by a single word is first determined based on the image resolution of the image to be processed and the font size in the table. Taking the image to be processed of 2453×3362 pixels as an example, the height and width of the small four-point font are approximately 32px, and the maximum area occupied by a single word is 1024. In actual applications, the image resolution and the font size in the table are dynamically adjusted according to the image to be processed, wherein the font size can be a parameter of the image to be processed, or can also be obtained by statistics through text detection results. After determining the maximum area occupied by a single word, a quadrilateral whose area is larger than the maximum area occupied by a single word is extracted from the fitted polygon. That is, quadrilaterals whose area is larger than the maximum area occupied by a single word are selected from the fitted polygon. For example, the OpenCV contour area calculation function contourArea is used to filter out quadrilaterals whose area is smaller than 1024 of the maximum area occupied by a single word. By adopting this embodiment, the interference of the text detected by the contour on the quadrilateral can be reduced, and the accuracy of cell detection can be improved.

[0041] After filtering out the quadrilaterals, further rectangular fitting is performed on the quadrilaterals to reduce the error caused by the short oblique lines in the subsequent horizontal and vertical line detection. Further optionally, in one embodiment, when performing rectangular fitting on the filtered quadrilaterals: for each quadrilateral, first take out the four vertices of the quadrilateral, and sort them in the order of upper left, upper right, lower right, and lower left according to the positions of the four vertices, that is, sort them in a clockwise direction, and the coordinates of the four vertices are (X [左上] , Y [左上] ), (X [右上] , Y [右上] ), (X [右下] , Y [右下] ), (X [左下] , Y [左下] ).

[0042] Recalculate the coordinates of the upper left and lower right vertices as follows: X [左上] ' = (X [左上] + X [左下] ) / 2, Y [左上] ' = (Y [左上] + X [右上] ) / 2, X [右下] ' = (X [右下] + X [右上] ) / 2, Y [右下] ' = (Y [右下] + Y [左下] ) / 2; Recalculate the coordinates of the upper right and lower left vertices as follows: X [右上] '= X [右下] ', Y [右上] ' = Y [左上] ', X [左下] ' = X [左上] ', Y [左下] ' = Y [右下] ', Among them, the four vertices (X [左上] ', Y [左上] '), (X [右上] ', Y [右上] '), (X [右下] ', Y [右下] '), (X [左下] ', Y [左下]') form a rectangle.

[0043] After the step of rectangle fitting, a rectangle set obtained from multiple rectangles can be obtained, and the cell is obtained through the rectangle set. Optionally, in one embodiment, the rectangle set can be directly used as a cell position set to obtain the cell; further optionally, in one embodiment, when the first cell coordinates are obtained according to the fitted rectangle: a blank image is generated according to the size of the image to be processed; the fitted rectangle is drawn into the blank image to obtain the second image; the line segments are extracted after edge detection of the second image to obtain a line segment set; the horizontal and vertical lines in the line segment set are identified to obtain a horizontal line set and a vertical line set; and the first cell coordinates are determined according to the intersection of the horizontal and vertical lines in the horizontal line set and the vertical line set. Wherein, if the image to be processed is perspective corrected, a blank image is generated according to the size of the image to be processed after perspective correction, and the above-mentioned fitted rectangle set is drawn into the blank image to obtain a table boundary image. In this embodiment, for the convenience of description and accurate distinction, the table boundary image at this location is defined as the second image. The inventor has found that the adjacent rectangles detected and fitted by contour detection do not completely overlap due to the influence of the roughness of the original table, and there is a certain gap, which causes some adjacent rectangles not to be detected. If the rectangle set is directly used as a cell, there will be detection errors. In order to solve this problem, in this embodiment, after the matrix set is drawn into the blank image to obtain the second image, edge detection and line segment extraction are performed again, and the intersection of the line segments is used to determine the cell coordinates. Specifically, the second image can be edge detected once using the edge detection function Canny of OpenCV, and the line segment extraction of the edge detection result can be performed using the HoughLinesP Huffman transform function to obtain a line segment set, and the horizontal and vertical lines therein are identified, and the cell coordinates are determined using the intersection of the horizontal and vertical lines.

[0044] Optionally, in one embodiment, how to identify the horizontal and vertical lines in the line segment set and obtain the steps of the horizontal and vertical line sets are described in detail as follows. Each line segment is punctuated by two endpoints. When identifying whether a line segment is a horizontal line or a vertical line, it is possible to first determine whether the ordinates of the two endpoints of the line segment are equal. If the ordinates of the two endpoints of the line segment are equal, the line segment can be determined to be a horizontal line; determine whether the horizontal coordinates of the two endpoints of the line segment are equal; if the horizontal coordinates of the two endpoints of the line segment are equal, the line segment can be determined to be a vertical line. If the horizontal and vertical coordinates of the two endpoints of the line segment are not equal, the slope of the line segment is calculated at this time. When the absolute value of the slope of the line segment is less than 1, the line segment is determined to be a horizontal line, and when the absolute value of the slope of the line segment is greater than 1, the line segment is determined to be a vertical line. Through this embodiment, on the one hand, the identification of horizontal and vertical lines in the line segment is realized, and on the other hand, the short oblique lines in the line segment set are processed to reduce the complexity of subsequent determination of cells based on the intersection of horizontal and vertical lines.

[0045] In this process, the inventors found that in the process of detecting the table, some lines may be broken or overlapped due to noise, printing quality or shooting angle, and these overlapping or disconnected lines will be identified as multiple independent lines, thereby causing some unnecessary false detections when identifying cells. Optionally, in one embodiment, when determining the first cell coordinates according to the intersection of the horizontal and vertical lines in the horizontal line set and the vertical line set, the horizontal line set and the vertical line set are first filtered once respectively to reduce the probability of the above false detection. The filtering step for the horizontal line set includes: finding the horizontal lines located in the same row in the horizontal line set to obtain the horizontal line row; traversing the horizontal line row to determine the relationship between adjacent horizontal lines in the row extension direction, wherein, if the current adjacent horizontal lines have overlapping parts or the gap is less than the first preset number of pixels, the adjacent horizontal lines are merged into a new horizontal line, and the next adjacent horizontal line is judged, and finally the horizontal line row is processed as a long horizontal line row. The filtering step for the vertical line set includes: finding the vertical lines in the same column in the vertical line set to obtain the vertical line column; traversing the vertical line column to determine the relationship between adjacent vertical lines in the column extension direction, wherein if the current adjacent vertical lines have overlapping parts or the gap is less than the second preset number of pixels, the adjacent vertical lines are merged into a new vertical line, and the next adjacent vertical line is judged, and finally the vertical line column is processed into a long vertical line column. Among them, the first preset number of pixels and the second preset number of pixels can determine a threshold value according to the line breakage situation, or an empirical value can be set, for example, the first preset number of pixels and the second preset number of pixels are both set to 10 pixels. After the above processing steps, the first cell is finally determined according to the intersection of the horizontal and vertical lines in the long horizontal line row and the long vertical line column. The recognition method of the table in the image provided by this embodiment can ensure that the lines are complete and continuous, thereby ensuring that the cell boundaries are clear, and the generated intersections are more accurate and non-redundant, so that the subsequent cell detection based on the intersection of the lines is more accurate. At the same time, the tables in the image to be processed may have slight misalignment or distortion due to scanning or shooting problems. By merging lines with slight gaps or overlaps, the algorithm can be more applicable to images of different qualities, ensuring that the table recognition process is not easily affected by minor image defects. It can also reduce the number of lines to be processed, simplify the calculation of intersections, and improve the algorithm's operating efficiency.

[0046] Furthermore, in one embodiment, the step of finding horizontal lines in the same row in the horizontal line set and obtaining the horizontal line row includes: calculating the ordinate of the midpoint of each horizontal line in the horizontal line set; sorting the horizontal lines in the horizontal line set according to the ordinate of the midpoint; treating the adjacent horizontal lines in the sorted horizontal line set whose ordinates of the midpoints differ by a third preset number of pixels as the horizontal lines in the same row, for example, the horizontal lines whose absolute value of the difference between the ordinates of the front and rear midpoints is less than 10 pixels are grouped together and regarded as the same row, which is defined as the horizontal line row. Similarly, the step of finding vertical lines in the same column in the vertical line set and obtaining the vertical line column includes: calculating the ordinate of the midpoint of each vertical line in the vertical line set; sorting the vertical lines in the vertical line set according to the ordinate of the midpoint; treating the adjacent vertical lines in the sorted vertical line set whose ordinates of the midpoints differ by a fourth preset number of pixels as the vertical lines in the same column, for example, the vertical lines whose absolute value of the difference between the ordinates of the front and rear midpoints is less than 10 pixels are grouped together and regarded as the same column, which is defined as the vertical line column.

[0047] When processing a horizontal line as a long horizontal line, Figure 2 This is a flow chart of processing a horizontal line row into a long horizontal line row in the method for recognizing a table in an image provided by the first embodiment of the present invention, such as Figure 2 As shown, the step of traversing the horizontal line row and determining the relationship between adjacent horizontal lines in the row extension direction includes: Step S21: sorting the horizontal lines in the horizontal line row according to the horizontal coordinates of the same endpoints of the line segments, for example, sorting the horizontal lines according to the horizontal coordinates of the left endpoints of each line segment, or sorting the horizontal lines according to the horizontal coordinates of the right endpoints of each line segment; Step S22: determining the relationship between the first horizontal line and the second horizontal line, wherein the initial values ​​of the first horizontal line and the second horizontal line are the first horizontal line and the second horizontal line in the horizontal line row, when the first horizontal line and the second horizontal line are in an inclusive relationship, that is, the first horizontal line completely covers the second horizontal line, or the second horizontal line completely covers the first horizontal line, executing step S23, when the first horizontal line and the second horizontal line are in a partially overlapping relationship, that is, the first horizontal line and the second horizontal line partially overlap, or when there is a gap between the first horizontal line and the second horizontal line in the row extension direction, and the gap is less than or equal to a first preset number of pixels, executing step S24, or when there is a gap between the first horizontal line and the second horizontal line in the row extension direction, and the gap is greater than the first preset number of pixels, executing step S26; Step S23: determining that the longer horizontal line between the first horizontal line and the second horizontal line is a new horizontal line, and executing step S25, that is, if the first horizontal line covers the second horizontal line, the first horizontal line is the new horizontal line, and if the second horizontal line covers the first horizontal line, the second horizontal line is the new horizontal line; Step S24: taking the abscissas of the two endpoints on the first horizontal line and the second horizontal line that are farthest apart as the abscissas of the new horizontal line, taking the average of the ordinates of the endpoints on the first horizontal line and the second horizontal line as the ordinate of the new horizontal line, that is, taking the abscissa of the left endpoint of the first horizontal line as the abscissa of the left endpoint of the new horizontal line, taking the abscissa of the right endpoint of the second horizontal line as the abscissa of the right endpoint of the new horizontal line, taking the average of the ordinates of the four endpoints on the first horizontal line and the second horizontal line as the ordinate of the new horizontal line, and executing step S25; Step S25: taking the new horizontal line as the first horizontal line, extracting the next horizontal line in the horizontal line row as the second horizontal line, that is, reassigning the current first horizontal line and the second horizontal line, and returning to step S22, wherein, when all the horizontal lines in the horizontal line row are extracted, that is, after all the horizontal lines in a horizontal line row are judged and compared, step S27 is executed; Step S26: storing the first horizontal line in the long horizontal line row, taking the second horizontal line as the first horizontal line, extracting the next horizontal line in the horizontal line row as the second horizontal line, that is, reassigning the current first horizontal line and the second horizontal line, and returning to step S22, wherein when all horizontal lines in the horizontal line row are extracted, executing step S27; Step S27: Output a long horizontal line.

[0048] When processing a vertical line column as a long vertical line column, Figure 3 This is a flow chart of processing a vertical line column into a long vertical line column in the method for recognizing a table in an image provided by the first embodiment of the present invention, such as Figure 3 As shown, the step of traversing the vertical line column and determining the relationship between adjacent vertical lines in the column extension direction includes: Step S31: sorting the vertical lines in the vertical line column according to the ordinate of the same endpoint of the line segment, for example, sorting the vertical lines according to the ordinate of the upper endpoint of each line segment, or sorting the vertical lines according to the ordinate of the lower endpoint of each line segment; Step S32: determining the relationship between the first vertical line and the second vertical line, wherein the initial values ​​of the first vertical line and the second vertical line are the first vertical line and the second vertical line in the vertical line column, when the first vertical line and the second vertical line are in an inclusive relationship, that is, the first vertical line completely covers the second vertical line, or the second vertical line completely covers the first vertical line, executing step S33, when the first vertical line and the second vertical line are in a partially overlapping relationship, that is, the first vertical line and the second vertical line partially overlap, or when there is a gap between the first vertical line and the second vertical line in the column extension direction, and the gap is less than or equal to the second preset number of pixels, executing step S34, or when there is a gap between the first vertical line and the second vertical line in the column extension direction, and the gap is greater than the second preset number of pixels, executing step S36; Step S33: determining that the longer vertical line between the first vertical line and the second vertical line is a new vertical line, and executing step S35, that is, if the first vertical line covers the second vertical line, the first vertical line is the new vertical line, and if the second vertical line covers the first vertical line, the second vertical line is the new vertical line; Step S34: taking the ordinates of the two endpoints on the first vertical line and the second vertical line that are farthest apart as the ordinate of the new vertical line, taking the average of the abscissas of the endpoints on the first vertical line and the second vertical line as the abscissa of the new vertical line, that is, taking the ordinate of the endpoint on the first vertical line as the ordinate of the upper endpoint of the new vertical line, taking the ordinate of the lower endpoint on the second vertical line as the ordinate of the lower endpoint of the new vertical line, taking the average of the abscissas of the four endpoints on the first vertical line and the second vertical line as the abscissa of the new vertical line, and executing step S35; Step S35: taking the new vertical line as the first vertical line, extracting the next vertical line in the vertical line column as the second vertical line, that is, re-assigning the current first vertical line and the second vertical line, and returning to step S32, wherein, when all vertical lines in the vertical line column are extracted, that is, all vertical lines in a vertical line row are judged and compared, executing step S37; Step S36: storing the first vertical line in the long vertical line column, taking the second vertical line as the first vertical line, extracting the next vertical line in the vertical line column as the second vertical line, that is, reassigning the current first horizontal line and the second horizontal line, and returning to step S32, wherein when all vertical lines in the vertical line column are extracted, executing step S37; Step S37: Outputting a long vertical line column.

[0049] Optionally, in one embodiment, after obtaining the long vertical line row and the long vertical line column, the step of determining the coordinates of the first cell according to the coordinates of the intersection of the horizontal and vertical lines in the long horizontal line row and the long vertical line column includes: comparing the horizontal and vertical lines in the long horizontal line row and the long vertical line column in pairs to determine whether there is an intersection, obtaining a vertex when there is an intersection, and writing the vertices into the vertex set in sequence according to the row extension direction and the column extension direction. Specifically, comparing the horizontal and vertical lines in the long horizontal line row and the long vertical line column in pairs to determine whether there is an intersection, taking the intersection as a vertex when there is an intersection, and recording the relationship between the adjacent vertices on the horizontal and vertical lines where the vertex is located from left to right and from top to bottom, that is, recording the next vertex of each vertex in the horizontal and vertical directions, and then putting them into the vertex set.

[0050] Figure 4 This is a flow chart of obtaining a first cell through a vertex set in a method for recognizing a table in an image provided by the first embodiment of the present invention, such as Figure 4 As shown, when traversing the vertex set and obtaining the first cell through the vertex set, the following steps are specifically performed: Step S410: obtaining a first vertex, wherein an initial value of the first vertex is the first vertex in the vertex set; Step S411: Determine whether there is an adjacent next vertex to the first vertex in the row extension direction, that is, when the row extension direction is the horizontal direction and the positive direction is from left to right, determine whether there is a vertex on the horizontal line where the first vertex is located that is adjacent to the first vertex on the right side of the first vertex. If not, execute step S412; if so, execute step S413; Step S412: taking the next vertex in the vertex set as the first vertex, that is, extracting the next vertex from the vertex set and assigning it to the first vertex, and returning to step S410; Step S413: taking the next vertex adjacent to the first vertex in the row extension direction as the second vertex, and taking the first vertex as the first vertex of the first cell; Step S414: Determine whether there is an adjacent next vertex in the column extension direction of the second vertex, that is, when the column extension direction is the vertical direction and the positive direction is from top to bottom, determine whether there is a vertex below the second vertex and adjacent to the second vertex on the vertical line where the second vertex is located. If not, execute step S415; if so, execute step S417; Step S415: Determine whether there is an adjacent next vertex to the second vertex in the row extension direction, that is, determine whether there is a vertex on the horizontal line where the second vertex is located that is adjacent to the second vertex on the right side of the second vertex. If not, execute step S412; if so, execute step S416; Step S416: taking the next vertex adjacent to the second vertex in the row extension direction as the second vertex, that is, reassigning a value to the second vertex, and returning to step S414; Step S417: taking the next vertex adjacent to the second vertex in the column extension direction as the third vertex, and taking the second vertex as the second vertex of the first cell; Step S418: Determine whether there is an adjacent previous vertex for the third vertex in the row extension direction, that is, determine whether there is a vertex on the horizontal line where the third vertex is located that is adjacent to the third vertex on the left side of the third vertex. If not, execute step S419; if so, execute step S421; Step S419: Determine whether there is an adjacent next vertex to the third vertex in the column extension direction, that is, determine whether there is a vertex on the vertical line where the third vertex is located below the third vertex and adjacent to the third vertex. If not, execute step S412; if so, execute step S420; Step S420: taking the next vertex adjacent to the third vertex in the column extension direction as the third vertex, that is, reassigning a value to the third vertex, and returning to step S418; Step S421: taking the previous vertex adjacent to the third vertex in the row extension direction as the fourth vertex, and taking the third vertex as the third vertex of the first cell; Step S422: Determine whether there is an adjacent previous vertex for the fourth vertex in the column extension direction, that is, determine whether there is a vertex on the vertical line where the fourth vertex is located that is above the fourth vertex and adjacent to the fourth vertex. If not, execute step S423; if so, execute step S425; Step S423: Determine whether there is an adjacent previous vertex for the fourth vertex in the row extension direction, that is, determine whether there is a vertex on the horizontal line where the fourth vertex is located that is adjacent to the fourth vertex on the left side of the fourth vertex. If not, execute step S412; if so, execute step S424; Step S424: taking the previous vertex adjacent to the fourth vertex in the row extension direction as the fourth vertex, that is, reassigning a value to the fourth vertex, and returning to step S422; Step S425: taking the previous vertex adjacent to the fourth vertex in the column extension direction as the fifth vertex, and taking the fourth vertex as the fourth vertex of the first cell; Step S426: Determine whether the fifth vertex is the same vertex as the first vertex, if not, execute step S427, if yes, execute step S429; Step S427: Determine whether there is an adjacent previous vertex for the fifth vertex in the column extension direction, that is, determine whether there is a vertex on the vertical line where the fifth vertex is located that is above the fifth vertex and adjacent to the fifth vertex; if not, execute step S412; if so, execute step S428; Step S428: taking the previous vertex adjacent to the fifth vertex in the column extension direction as the fifth vertex, that is, reassigning a value to the fifth vertex, and returning to step S426; Step S429: Determine the coordinates of the first cell using the coordinates of the first vertex of the first cell, the second vertex of the first cell, the third vertex of the first cell and the fourth vertex of the first cell, a total of four vertices.

[0051] After determining the coordinates of a first cell, continue to return to step S412, traverse the vertex set, all vertices in the vertex set are traversed, and all first cells on the image to be processed are obtained. Each first cell is defined by the coordinates of four vertices, and the coordinate set of all first cells in the table in the image to be processed is obtained.

[0052] Step S102: extracting a cell image from the image to be processed using the cell coordinates.

[0053] After the cell coordinates are obtained through the above step S101, in step S102, the image to be processed after perspective correction is taken, and the cell coordinates are used to perform slice processing on the corrected image to be processed. Each cell coordinate can be extracted into a cell image, and finally all cell coordinates are extracted into a set of cell images.

[0054] Step S103: Recognize the text in the cell image.

[0055] After obtaining the cell image set through the above step S102, in this step S103, the text in each cell image is recognized, and specifically, an OCR recognition method can be adopted, or an intelligent model can be adopted to perform image recognition processing.

[0056] The inventors have found that when a cell image includes multiple text blocks, the spaces between the text blocks and the first character, the last character, etc. in the cell are easily lost during recognition. In order to solve this technical problem, optionally, in one embodiment, the step of recognizing text in the cell image includes: detecting a second text area in the cell image; determining a second text area located in the same row among the multiple detected second text areas to obtain a text area row; determining the upper and lower edges of the text area row extending in the row extension direction based on the vertices of the second text area; determining the left and right edges of the text area row based on the boundary line of the cell in the column extension direction; determining the target area using the upper, lower, left and right edges of the text area row; recognizing the text in the target area to obtain the text in the cell image.

[0057] Specifically, when identifying text in a cell image, a text detection model is first used to detect a text area from the cell image, which is defined as a second text area in this embodiment. Then, the second text area located in the same row is found from these second text areas. For example, the two second text areas can be sorted according to the average of the ordinates of the four vertices of each second text area. If the average of the ordinates of the two second text areas differ by less than half a font size, the two second text areas are placed in the same row to obtain a text area row. For each text area row, the vertices of each second text area in the text area row are used to determine the upper and lower edges of the text area row extending in the row extension direction, so as to merge the second text areas in the same text area row; the left and right edges of the text area row are determined by the boundary line of the cell in the column extension direction, that is, the left and right boundary lines, so as to widen the second text area in the same text area row to the left and right. For example, each text area row is traversed, and the ordinate mean of the upper left and upper right vertex sets of each second text area row in the text area row is calculated as the ordinate top_y of the upper edge of the current text area row, and the ordinate mean of the lower left and lower right vertex sets of each second text area row is used as the ordinate bottom_y of the lower edge of the current text area row, and the left and right boundary lines of the cell are taken as the coordinates of the left and right boundary lines of the text row as the horizontal coordinates left_x and right_x of the left line of the current text area row, respectively. When the upper, lower, left and right lines of the text area row are used to determine the target area, the above coordinates top_y, bottom_y, left_x, and right_x are used to recompose the four vertex coordinates of the target area [(left_x, top_y), (right_x, top_y), (right_x, bottom_y), (left_x, bottom_y)] to obtain the target area, and then the text in the target area is recognized to obtain the text in the cell image. Specifically, an open source text recognition model can be used, and characters commonly used in the corresponding industry but not in the open source model are added for training, and then the trained model is used to process the target area to obtain the text content. For example, if the table in the image to be processed is a quota table, then characters commonly used in the construction industry are added to the training set to improve the accuracy of text recognition. For a cell image with multiple target areas, the text contents recognized in the multiple target areas are spliced ​​in order from top to bottom to obtain the complete content in the cell.

[0058] The method for recognizing tables in images provided in this embodiment is used to obtain a target area by merging and widening the text areas of the same line of text in a cell. This target area is used as the target for subsequent text recognition processing. This can not only provide spatial assistance for identifying "spaces" during subsequent text recognition, but also avoid partial or even complete omission of the first character or the last punctuation mark, thereby providing an accurate detection area for subsequent text recognition processing and improving the accuracy of text recognition.

[0059] The inventors further discovered that when there is a background color in the cell and the text consists of only a single character (such as l, -, t, etc.), and the distinction between the character color and the background color is not particularly obvious, it is easy to fail to detect the text area when detecting the cell image, resulting in missed detection. In order to improve the detection rate and accuracy of text detection in this situation, optionally, in one embodiment, the step of identifying the text in the cell image also includes: when the second text area cannot be detected, obtaining preset parameters corresponding to the typeset features of the table; determining the upper and lower borders of the suspected text area in the cell image according to the preset parameters; determining the left and right borders of the suspected text area according to the boundary line of the cell in the column extension direction; and determining the target area using the upper, lower, left and right borders of the row of the suspected text area.

[0060] Specifically, when the second text area is not detected in the cell image, a suspected text area can be determined in the cell image through the typesetting features of the table. For example, by utilizing the typesetting habit of centering the text in a table, the top_y and bottom_y are expanded upward and downward by a certain number of pixels at the center of the cell image according to the maximum height of a character. The left and right horizontal coordinates of the cell are still used as left_x and right_x. In this way, a suspected text area can be filled in as the target area. Whether there is text in the target area can be processed and judged through subsequent text recognition steps to avoid missed detections in text recognition and further improve the accuracy of table recognition.

[0061] Step S104: constructing a table in the image to be processed using multiple cell coordinates.

[0062] In step S104, the cell coordinate set obtained in step S101 can be used to construct a table in the image to be processed.

[0063] Step S105: Output the text in the cell image to the table.

[0064] In step S105 , the text in each cell graphic is output to its corresponding cell, that is, output to the table.

[0065] In the method for recognizing a table in an image provided by this embodiment, when recognizing a table in an image to be processed, the cell coordinates in the image to be processed are first detected, and then the cell image is extracted from the image to be processed using the cell coordinates, and the text in the cell image is recognized, and finally the table in the image to be processed is constructed using all the cell coordinates, and the text in the cell image is output to the table accordingly. The method for recognizing a table in an image provided by this embodiment can improve the accuracy of recognizing a table in an image.

[0066] Optionally, in one embodiment, the method for recognizing a table in an image further includes: before the step of detecting cells in the image to be processed, performing overall perspective correction on the image to be processed; after the step of extracting a cell image in the image to be processed using cell coordinates, recognizing text outside the cell image; after the step of constructing a table in the image to be processed using multiple cell coordinates, outputting the text outside the cell image outside the table.

[0067] Specifically, when perspective correction is performed on the image to be processed, the entire image to be processed is corrected, that is, not only the table part is corrected, but the entire image to be processed is corrected. Through the overall correction method, it is convenient to extract and recognize the text of the work content, unit, notes, etc. near the table. That is to say, after extracting the cell image, in addition to recognizing the text of the cell image, the text outside the cell image is also recognized, that is, the content near the table, and after constructing the table in the image to be processed, the recognized text outside the cell image is output outside the table.

[0068] The method for recognizing a table in an image provided by this embodiment is helpful for extracting the content outside the table and the vicinity based on the perspective correction of the entire image, thereby improving the integrity of table recognition.

[0069] Further optionally, in one embodiment, the step of performing overall perspective correction on the image to be processed includes: performing grayscale processing and binarization processing on the image to be processed in sequence to obtain a first image; identifying the quadrilateral with the largest area in the first image; extending the four sides of the quadrilateral with the largest area to the boundary of the first image; translating the four extended sides outward so that the extension line of each side can pass through at least one corner of the first image; calculating the four intersection points of the four translated sides; and performing perspective correction on the portion of the image to be processed defined by the four intersection points.

[0070] Specifically, Figure 5 This is a schematic diagram of perspective correction in the method for recognizing a table in an image provided by the first embodiment of the present invention, as shown in FIG. Figure 5As shown in the figure, the first image is the first image obtained after grayscale processing and binarization processing are performed on the image to be processed in sequence. After identifying the quadrilateral with the largest area in the first image, its four sides are extended to the boundary of the first image, as shown in the second image in the figure; then the four sides are translated outward, and the extension line of each side can at least pass through one corner of the first image, as shown in the third image in the figure. The four sides after translation have four intersection points. The four intersection points are calculated to obtain the four vertices that need to be transformed. The OpenCV perspective transformation function warpPerspective is used to convert the contents of the four vertices in the original image to be processed into the new target image, thereby obtaining a corrected image, as shown in the fourth image in the figure.

[0071] Figure 6 This is a flowchart of identifying the largest quadrilateral in the method for identifying a table in an image provided in the first embodiment of the present invention. Further, optionally, as Figure 6 As shown, in one embodiment, the step of identifying the quadrilateral with the largest area in the first image includes: Step S11: setting parameter values ​​of the contour search function and parameter values ​​of the polygon fitting function.

[0072] For example, the findContours function of OpenCV is used to find all the contours in the image from the binary image; polygon fitting is performed on each contour set found by using the OpenCV polygon fitting function approxPolyDP. The initial values ​​of the parameters of the contour search function and the polygon fitting function can be set according to empirical values, and different parameter values ​​are set each time this step is executed.

[0073] Step S12: Detect the contour boundary line in the first image by using the contour search function, perform polygon fitting on the contour boundary line in the first image by using the polygon fitting function, and input the quadrilateral with the largest area among the fitted polygons into the quadrilateral set.

[0074] Step S13: Determine whether the preset loop end condition is met. If the loop end condition is not met, return to step S11. If the loop end condition is met, execute step S14.

[0075] For example, the loop end condition is set to the number of polygon fittings being greater than a preset number.

[0076] Step S14: outputting the quadrilateral with the largest area in the first image according to the quadrilateral set.

[0077] By adopting the method for recognizing tables in images provided in this embodiment, by repeatedly using different parameter values ​​for contour search and polygon fitting, the influence of noise in the image and the degree of distortion of the table lines themselves on finding the largest quadrilateral can be reduced, thereby improving the influence of finding the largest quadrilateral.

[0078] Optionally, in one embodiment, the table in the image to be processed includes a portion in which rows are distinguished by alternating different background colors. For example, the image to be processed includes rows distinguished by alternating a first background color and a second background color. In this case, the step of identifying the contour boundary lines in the image to be processed includes: converting the foreground text in the image to be processed into black, converting the first background color into black, and converting the second background color into white to obtain a first converted image; converting the foreground text in the image to be processed into black and converting all the backgrounds into white to obtain a second converted image; detecting the contour boundary lines in the first converted image and the second converted image, wherein polygon fitting is performed on the contour boundary lines in the first converted image and the second converted image, respectively, quadrilaterals are filtered out from the fitted polygons, and rectangle fitting is performed on the filtered quadrilaterals, respectively, to obtain a set of rectangle sets; the step of drawing the fitted rectangles into the blank image includes: drawing both sets of rectangle sets into the blank image.

[0079] Specifically, the image to be processed after perspective correction can be first subjected to grayscale processing, and then the grayscale image can be binarized. By changing the foreground text and the first background color to black and the second background color to white, a first conversion image is obtained. The first conversion image is a binarized image, and the alternating rows of different backgrounds are converted into alternating rows of black and white. By detecting the contour boundary line of the first conversion image, the alternating black and white boundary line can be accurately detected; by converting the foreground text to black and the entire background to white, a second conversion image is obtained. The second conversion image is also a binarized image, and the boundary formed by the alternation of different background colors is removed. By detecting the contour boundary line of the second conversion image, the boundary lines of other areas except the rows that are not distinguished by the alternating background colors can be accurately detected. The boundary lines obtained from the two conversion images constitute the contour boundary lines in the image to be processed.

[0080] After obtaining two sets of boundary lines, polygon fitting is performed respectively, and then quadrilaterals are filtered out from the fitting results respectively, and rectangle fitting is further performed, and finally a set of rectangle sets are obtained respectively, that is, a set of rectangle sets is obtained through each conversion image, and finally both sets of rectangle sets are drawn into the blank image to obtain a merged image of the two sets of rectangle sets, that is, the second image. The intersection of the line segments in the second image distinguishes all the cells, and the subsequent steps of extracting line segments, identifying the horizontal and vertical lines therein, and determining the cells using the intersection of the horizontal and vertical lines can be referred to above, and will not be repeated here.

[0081] Using the method for recognizing tables in images provided in this embodiment, when the image to be processed contains a table in which rows are distinguished by alternating different background colors, the image to be processed is converted into a converted image that highlights contour boundary lines at different positions, and the boundary lines are extracted and then merged into the same image, thereby improving the accuracy of contour boundary line recognition.

[0082] Figure 7-Figure 12 This is a recognition effect diagram of table recognition in an image provided by the first embodiment of the present invention, such as Figure 7 As shown, optionally, in one embodiment, the first background color is gray and the second background color is white. Figure 8 As shown, the foreground text in the image to be processed is converted to black, the first background color is converted to black, and the second background color is converted to white. The step of obtaining the first converted image includes: after grayscale processing of the image to be processed, the pixel values ​​of the points with pixel values ​​less than 240 in the image to be processed are all set to 0, and the pixel values ​​of the points greater than or equal to 240 are all set to 255, that is, Figure 8 The gray background will turn black as shown in Fig. 9 As shown, the foreground text in the image to be processed is converted to black, and the entire background is converted to white. The step of obtaining the second converted image includes: after grayscale processing the image to be processed, finding the minimum pixel value of the points in the first background color area, and setting the pixel values ​​of all points in the image to be processed whose pixel values ​​are greater than the minimum pixel value to 255, that is, Fig. 9 The gray background will turn white as shown in the figure.

[0083] Continue to refer Figure 8 and Fig. 9 , turn the gray background to black to get the first conversion image as Fig.10 As shown, the gray background is changed to white to obtain the second conversion image as shown in Fig.11 As shown, the first conversion image and the second conversion image are respectively subjected to contour detection, polygon fitting, and rectangle fitting. That is, the step of "contour rectangularization" is performed to obtain two sets of rectangle sets. The two sets of rectangle sets are merged and redrawn to obtain the second image, as shown in FIG. Fig.12 shown.

[0084] It should be noted that Figure 7-Figure 12 The image to be processed is only used to schematically illustrate the basic outline and lines of the table, as well as the effect of obtaining a second image from the image to be processed. The text content in the table neither serves as any explanatory or schematic function nor constitutes a limitation on the solution.

[0085] Figure 13-Figure 16 Another recognition effect diagram of table recognition in an image provided by the first embodiment of the present invention is shown as follows: Fig.13As shown, optionally, in one embodiment, the table in the image to be processed also includes some cells without demarcation markers, and these cells are usually set in other cells with demarcation markers, such as lines, etc. In this embodiment, the cells without demarcation markers are defined as second cells, and the cells with demarcation markers are defined as first cells. The image to be processed targeted by this embodiment includes a first cell and a second cell, wherein the second cell is located within the first cell and has no demarcation marker. In this case, the step of identifying the contour boundary line in the image to be processed includes: grayscale processing the image to be processed to obtain a grayscale image; detecting the contour boundary line in the grayscale image; the step of detecting the cell in the image to be processed also includes: identifying a preset character area in the first cell according to the text features of the preset characters; calculating the midline of each preset character area in the first cell to obtain a midline set corresponding to the first cell; and determining the second cell according to the midline set.

[0086] Specifically, if Fig.13 As shown, the table in the image to be processed includes some special columns, and there are no demarcation marks between the rows in these columns. When identifying the contour boundary lines in the image to be processed, it is grayscale processed to obtain a grayscale image, and then the contour boundary lines in the grayscale image are detected. When detecting the cells in the image to be processed based on the contour boundary lines detected here, only the first cell in the image to be processed can be detected. The specific detection steps can be referred to above and will not be repeated here. On this basis, when further detecting the second cell without a demarcation mark in the first cell, first identify the preset character area in the first cell based on the text features of the preset characters. The preset characters are determined according to the specific style of the table type, and these character areas are identified through the text features of these characters. The preset characters are decimals, such as Fig.14 As shown, the grayscale image is first subjected to an OCR recognition to obtain multiple text regions, and then, as Fig.15 As shown in , the text features of numbers are used to filter out the pure decimal area from the text area, that is, the preset character area. Then, as Fig.16 As shown, the midline of each preset character area in the first cell is calculated to obtain a midline set corresponding to the first cell, and the midline of the preset character area is used to obtain the second cell.

[0087] Using the method for recognizing tables in images provided by this embodiment, when there are rows without boundary markers in the image to be processed, these rows are set in the first cell, and the text content in these rows is specific characters, the text area in the cell image corresponding to the first cell is first identified, and the area corresponding to the specific characters, that is, the preset character area, is found from these text areas. The center line set of these preset character areas is used to identify the rows without boundary markers to obtain the second cell, thereby realizing the recognition of the rows without boundary markers and improving the completeness and accuracy of table recognition in the image.

[0088] It should be noted that Figure 13-Figure 16 The image to be processed is only used to schematically illustrate the basic outline, lines and content type of the table, such as whether the content is numbers or Chinese characters, and to illustrate the line set obtained from the image to be processed. The specific content of the text in the table, such as what the specific Chinese characters are and what the specific numbers are, does not serve any explanatory purpose nor constitute a limitation on the solution.

[0089] Optionally, in one embodiment, first cells in the same row are processed uniformly. Fig.17 This is a flow chart of determining a second cell according to a centerline set in a method for recognizing a table in an image provided by Embodiment 1 of the present invention. Fig.18 and Fig.19 This is another recognition effect diagram of table recognition in an image provided by the first embodiment of the present invention, such as Figures 16 to 19 As shown, for each first cell in the same row, the step of determining the second cell according to the center line set includes: Step S50: extracting a first edge line, wherein the initial value of the first edge line is the first edge line of the first cell in the row extension direction, that is, the upper edge line of the first cell. Fig.18 Display the borders of the cell (the first cell) detected in the image to be processed.

[0090] Step S51: determining a center line adjacent to the first side line in the center line set to obtain a reference center line; Step S52: determine whether the reference center line is the last center line in the center line set, if not, execute step S53, if yes, execute step S56; Step S53: Calculate the distance between the first sideline and the reference centerline to obtain a half-line distance, and translate the reference centerline in a direction away from the second sideline by half the line distance to obtain the second sideline; Step S54: determine the first edge line, the second edge line and the portion of the first cell intercepted by the first edge line and the second edge line as the second cell, that is, calculate the upper and lower edges of the implicit cell (the second cell) according to the position of the unbounded cell where the center line is located.

[0091] Step S55: taking the second edge line as the first edge line, that is, re-assigning the first edge line, and returning to step S50; Step S56: Determine the first edge line, the second edge line of the first cell in the row extension direction, and the portion of the first cell intercepted by the first edge line and the second edge line as the second cell, such as Fig.19 shown.

[0092] It should be noted that Fig.18 and Fig.19 The image to be processed is only used to schematically illustrate the basic outline, lines and content type of the table, such as whether the content is numbers or Chinese characters, and to illustrate the use of the center line set and the edge line of the first cell to obtain the edge line of the second cell. The specific content of the text in the table, such as what the specific Chinese characters are and what the specific numbers are, does not serve any explanatory purpose nor constitute a limitation on the solution.

[0093] Optionally, in one embodiment, the step of identifying a preset character area in a first cell according to text features of preset characters includes: detecting a text area in an image to be processed to obtain a first text area; cross-judging the first text area and the first cell to determine the first text area in the first cell; and for the first text area in the first cell, filtering out the first text area containing the preset characters according to the text features of the preset characters to obtain the preset character area in the first cell.

[0094] Specifically, the text area in the image to be processed is first detected. In this embodiment, for the convenience of description and distinction, it is defined as the first text area, and then a cross judgment is made with the first cell to determine the first text area in the first cell. Optionally, the identified first text area can be arranged in rows first, that is, the first text area located in the same row is found according to the coordinates of the first text area, and then the first text area in the same row and the first cell in the same row are cross-judged to accurately find the first text area in the first cell. At the same time, when the first text area included in the first cell in the same row is also located in the same row, the horizontal edge line of the second cell in a first cell in the row is extended to obtain the second cell in other first cells. In addition, after obtaining the first text area, screening is performed according to the text features of the preset characters to obtain the first text area containing the preset characters, that is, the preset character area in the first cell.

[0095] Embodiment 2 The embodiment of the present invention provides a method for recognizing a table in an image. The method can be applied to the recognition of a quota table. By using the method, the recognition accuracy of the quota table in the image can be improved. Specifically, Fig. 20This is a flow chart of a method for recognizing a table in an image provided by Embodiment 2 of the present invention. Fig.11 As shown, the method for recognizing a table in an image provided by this embodiment includes the following steps.

[0096] Among them, in finding the exact position of each cell, there are two situations. One is the recognition of tables with gray and white backgrounds; the other is the recognition of tables with no upper and lower dividing lines for cells. The number of cells is increased by adding boundary separations to the cost, personnel and equipment detail areas in the quota table that have no upper and lower dividing lines.

[0097] Each of the above parts will be described in detail below.

[0098] 1. Image Correction (1) First, convert the original color image containing the quota table into a grayscale image (2) The grayscale image is then processed by Gaussian blur and binarization to obtain a binary image. (3) Use OpenCV's findContours function to find all contours in the image from the binary image.

[0099] The steps (1) to (3) above are used to complete the steps of contour detection on the image.

[0100] (4) Perform polygon fitting on each contour set found using the OpenCV polygon fitting function approxPolyDP and find the quadrilateral with the largest area In order to reduce the influence of noise in the image and the distortion of the table itself on the search for the largest quadrilateral, steps (2) to (4) are repeated, and the precision and parameters of each function are set to different values. The largest quadrilateral is finally obtained by using multiple searches for the largest quadrilateral.

[0101] (5) In order to correct the orientation of the entire image, the four sides of the largest quadrilateral found are extended to the edge of the image, and then the four sides are translated outward so that the extension line of each side can pass through at least one corner of the image, that is, through the first vertex on the corresponding adjacent edge of the image.

[0102] (6) Calculate the intersection points of the four extended and translated edge lines to obtain the four vertices that need the transformation matrix (7) The contents of the four vertices in the original image obtained in step (6) are converted into the new target image through the OpenCV perspective transformation function warpPerspective, thereby obtaining a corrected image.

[0103] 2. Cell Detection Part There are various styles of quota tables. What has the greatest impact on cell detection is that there are no obvious dividing lines between rows in some areas, some rows are distinguished by gray or white backgrounds, and some rows have no dividing markers at all. In this embodiment, the main body directly divides them into two situations for processing.

[0104] 2.1 When there is a gray or white background between rows (reference Figure 7 ) (1) Convert the corrected color image to a grayscale image, convert the foreground text and background of the gray background row to black, and perform a binarization process on the grayscale image, set all points with pixel values ​​less than 240 to black 0, and set all points with pixel values ​​greater than 240 to white 255, and finally obtain an image A (first conversion image) with black and white rows in the table; convert the background of the gray background row to white, and perform a statistical analysis on the grayscale image to obtain the threshold range of the gray background; perform another binarization process, set all points with pixel values ​​greater than the minimum value of the gray background to white 255, and keep the others unchanged; obtain image B (second conversion image) (2) Perform contour detection on the binary images A and B respectively, and filter out quadrilaterals from the detection results. For the specific process, please refer to the relevant content in the first part.

[0105] In order to reduce the text interference detected by contour, the contour area calculation function contourArea of ​​OpenCV is used to filter out quadrilaterals with an area smaller than the maximum area occupied by a single word, and two sets of quadrilateral vertex sets are obtained. The area occupied by a single word in the scanned image of the quota table printout is related to the image resolution and font size. Taking a scanned image of 2453×3362 pixels as an example, the height and width of the small 4-point font are about 32px, and the maximum area occupied is 1024; so quadrilaterals smaller than 1024 can be filtered out. In actual applications, this value will be dynamically adjusted according to the image resolution and the font size in the table; the font size is obtained based on the statistics of the text detection results.

[0106] The two sets of quadrilateral vertex sets are converted to rectangles by fitting, in order to reduce the error caused by short oblique lines in line continuity detection during subsequent horizontal and vertical line detection. The specific implementation steps have been described in detail in the first embodiment, and will not be repeated here.

[0107] (3) Generate a blank image according to the size of the corrected image, and draw the edges of the two sets of rectangles calculated in step (2) into the blank image, so as to obtain a merged image of the rectangle edges detected for the binary images A and B, i.e., the table edge map. Their intersection points perfectly distinguish all the cells.

[0108] (4) Due to the roughness of the original table and the influence of the rectangle fitting in step (2), the adjacent rectangle edges do not completely overlap and there is a certain gap. Use the edge detection function Canny of OpenCV to perform an edge detection on the table image drawn in step (3), and then use the HoughLinesP function of HoughLinesP to extract line segments from the edge detection results to obtain a line segment set; calculate the slope based on the two end points of the line segment, and the two vertices with equal y values ​​or absolute values ​​less than 1 are classified into the horizontal line set, and the two vertices with equal x values ​​or absolute values ​​greater than 1 are classified into the vertical line set.

[0109] After sorting the horizontal line set by the y coordinates of their respective midpoints, the horizontal lines whose absolute value of the y coordinate difference between the front and rear midpoints is less than 10 pixels are grouped together and regarded as the same row; after sorting each row of horizontal line sets by the x coordinates of the left vertex of the line segment, there will be three types of relationships between the front and rear line segments: inclusion, partial overlap, and gap; if there is an inclusion relationship, take the longest line segment and continue to compare and judge with the next one; if there is a partial overlap, take the x coordinates of the leftmost and rightmost vertices of the front and rear line segments as the x coordinate, and the average of the y coordinates as the y coordinate, merge them into a new line segment and continue to compare and judge with the next one; if there is a gap between the two line segments, when the absolute value of the x coordinate difference between the right and left vertices of the front and rear line segments is less than 10 pixels, take the x coordinates of the leftmost and rightmost vertices of the front and rear line segments as the x coordinate, and the average of the y coordinates as the y coordinate, merge them into a new line segment and continue to compare and judge with the next one; finally, several rows of the longest horizontal line segment sets are obtained.

[0110] After sorting the vertical line set by the x-coordinates of their respective midpoints, the vertical lines whose absolute value of the x-coordinate difference between the front and rear midpoints is less than 10 pixels are grouped together and considered as the same column; after sorting each row of vertical line sets by the y-coordinates of the upper vertices of the line segments, there will be three types of relationships between the upper and lower line segments: inclusion, partial overlap, and gap; if there is an inclusion relationship, take the longest line segment and continue to compare and judge with the next one; if there is a partial overlap, take the y-coordinates of the upper and lower vertices of the upper and lower line segments as the y-coordinate, and the average of the x-coordinates as the x-coordinate, merge them into a new line segment and continue to compare and judge with the next one; if there is a gap between the two line segments, when the absolute value of the y-coordinate difference between the upper and lower vertices of the front and rear line segments is less than 10 pixels, take the y-coordinates of the upper and lower vertices of the upper and lower line segments as the y-coordinate, and the average of the x-coordinates as the x-coordinate, merge them into a new line segment and continue to compare and judge with the next one; finally, several rows of the longest vertical line segment sets are obtained.

[0111] Realize the merging of adjacent horizontal and vertical lines.

[0112] (5) The longest horizontal line segment set obtained in step (4) above is compared with the longest vertical line segment set to determine whether there is an intersection. If there is an intersection, a vertex is obtained and the relationship between the adjacent vertices on the horizontal and vertical line segments where the vertex is located is recorded from left to right and from top to bottom. That is, the next vertex of the current vertex in the horizontal and vertical directions is recorded and then put into the table vertex set.

[0113] To traverse the table vertex set, the basic idea is as follows: Take a vertex vertex and determine whether there is a next vertex HNext in the horizontal direction; If the vertex HNext exists, take the vertex HNext and determine whether there is a next vertex VNext in the vertical direction of the vertex HNext; If the vertex VNext does not exist, then take the next vertex HNext' of the vertex HNex in the horizontal direction, until the next vertex VNext is found in the vertical direction of HNext'; If there is a vertex VNext, take the vertex VNext, and then determine whether there is a previous vertex HPre in the horizontal direction of the current vertex; If the vertex HPre does not exist, then take the next vertex VNext' of the vertex VNext in the vertical direction until the previous vertex HPre is found in the horizontal direction of VNext'; If there is a vertex HPre, take the vertex HPre and then determine whether there is a previous vertex VPre in the vertical direction of the vertex HPre; If there is no vertex VPre, then take the previous vertex HPre' of the vertex HPre in the horizontal direction, until a previous vertex VPre is found in the vertical direction of HPre'; If there is a vertex VPre, and the vertex VPre is the same vertex as the vertex vertex, then a minimum cell is obtained. If the vertex VPre is not the same vertex as vertex vertex, continue to search for the previous vertex VPre' in the vertical direction of VPre until VPre' at the same point as vertex is found, then the minimum cell exists; otherwise, it does not exist.

[0114] After the above steps are processed, the minimum rectangle set is obtained, that is, the coordinate set of all cells in the quota table, that is, the cell set.

[0115] 2.2 When there are no demarcation markers between rows (refer to Figure 8 and Fig.10 ) The situation where there are no dividing markers between rows mainly occurs in the expense part and the personnel and equipment part of the quota table; the situation where there are line breaks within cells mainly occurs in the personnel and equipment name column. The inventors have found that when the text detection model detects text boxes (text areas), it is easy to lose the initial Chinese characters, the ending punctuation marks and individual letters, but it has a good effect on decimal detection and rarely misses. This embodiment takes advantage of this feature, and according to the text detection model's recognition of the position of the digital area and the upper and lower boundaries of the cells detected in the expense area and the personnel and equipment area, it combines calculations to reorganize the position of the row boundaries. The specific implementation steps are as follows: (1) First, perform text detection and text recognition on the corrected image in the first part (2) Convert the color image corrected in step 1 into a grayscale image (3) Perform contour detection on the grayscale image, filter out quadrilaterals from the detection results, and fit to generate a set of rectangles. (For the specific process, please refer to the relevant description in Section 2.1 above) (4) Generate a blank image according to the size of the rectified image, draw the rectangle set on the blank image, and obtain the table boundary map. The reason why the rectangle set is not directly used as the cell position set here is that some adjacent rectangles obtained by contour detection and fitting will not be detected, but can be reassembled through adjacent edge and vertex calculations.

[0116] (5) Use OpenCV's edge detection function Canny and HoughLinesP to detect the horizontal and vertical lines of the table drawn in step (3) and merge adjacent horizontal and vertical lines.

[0117] (6) Calculate the intersection points, traverse them, and calculate the minimum set of rectangles, that is, the set of cells. (For the specific process, please refer to the relevant description in Section 2.1 above) (7) Calculate the mean y' of the y coordinates of the four vertices of the text box obtained in step (1) according to its coordinates, and after sorting, group the text boxes whose absolute value of the difference between the mean y' of the two text boxes before and after is less than half the character height into one group and regard them as a row; sort the text boxes in each row according to the leftmost x coordinate, and finally obtain the result of the image text box recognition result after dividing into rows.

[0118] The cell coordinate set is cross-judged with the above text box rows to determine which text boxes are contained in each cell. The specific judgment rule is that the x coordinate of the middle point of the text box is between the left and right edges of the cell, and the y coordinate of the middle point of the text box is between the upper and lower edges of the cell.

[0119] According to the text features and cell coordinate positions contained in the cells, the cost area and the personnel, materials and machinery area are identified, and the contents that are pure decimals or decimals surrounded by parentheses are extracted from these two areas (decimals surrounded by parentheses in the quota table indicate that the current material is the main material).

[0120] (8) Calculate the average of the y coordinates of all text boxes in each row whose text content is a decimal. The calculated average is regarded as the y coordinate of the middle line of the current row, recorded as line_middle_y, and included in all cells in the same row. After this calculation, the cells in the cost and personnel area contain a line_middle_y set. (9) Traverse the cells in the expense and personnel and equipment areas, and traverse the line_middle_y set under each cell in ascending order (i.e., from top to bottom). The distance from the y coordinate top_y of the top edge of the cell to the first line_middle_y is half the height of the first row of expense or personnel and equipment cells in the current cell. Extend the same distance downward to get the bottom line position of the first row of personnel and equipment cells. Treat the bottom line position of the first row of expense or personnel and equipment as the top line position of the second row, and calculate repeatedly to get the top and bottom coordinates of each row of expense or personnel and equipment cells. Combine the x coordinates of the left and right edges of the cells in the expense and personnel and equipment areas to get the position of each expense and personnel and equipment cell. When traversing to the last line_middle_y under each cell, directly take the y coordinate bottom_y of the bottom edge of the current cell as the bottom line y coordinate of the last expense or personnel and equipment cell.

[0121] After processing the above steps, it is possible to complete the distinction between the cost area and the personnel and equipment area rows in the quota table when there are no dividing markers, providing a basis for subsequent detailed text detection and extraction in cells, and preventing cells from being mistakenly divided into the next row when there is a line break in the cell.

[0122] Part 3: Text detection in cells (1) Take the corrected image in the first part, slice the corrected image with the quota table cell set obtained in the second part, take the cell image set, traverse the cell image set, and use the open source text detection model to detect the text box (2) If there are multiple text boxes in a cell, they are sorted by the mean y' of the vertices of each text box. If the absolute value of the difference between the mean y' of the two regions is less than half the font size, they are placed in the same row. Each row is traversed to calculate the mean y coordinate of the upper left and upper right vertex sets of each text box as the y coordinate top_y of the upper line of the current row, and the mean y coordinate of the lower left and lower right vertex sets of each text box as the y coordinate bottom_y of the lower line of the current row.

[0123] To prevent the text detection model from losing or partially losing the first character or the last punctuation in some scenarios, the left and right x coordinates of the same line of text are directly extended to the left and right boundaries of the cell. That is, the left and right boundaries of the cell are directly used as the left_x and right_x coordinates of the text line.

[0124] According to the above top_y, bottom_y, left_x, right_x, the four vertex coordinates of each line of text box are recomposed [(left_x, top_y,), (right_x, top_y), (right_x, bottom_y), (left_x, bottom_y)] to obtain a target area.

[0125] (3) When there is a background color in the cell, and the text only contains a single character (such as l, -, t, etc.), and the character color is not particularly different from the background color, the text detection model may not detect the text box. In order to improve the detection rate of text detection in this case, this embodiment adopts a remedial method: when there is no text box in the cell, the text of the quota table is centered vertically, and a text box is added in the center of the cell, using the maximum height of a character + 4 pixels of expansion up and down as the top_y and bottom_y coordinates, and the left and right x coordinates of the cell are still used as the left_x and right_x. The left and right x coordinates of the cell are used as the left_x and right_x. Whether there is text content is handed over to the subsequent text recognition model for processing and judgment, and the text detection rate in the cell is further improved.

[0126] (4) Characters commonly used in the construction industry but not in the open source model are added to the open source text recognition model, and the above target area is recognized and processed to obtain the text content; if there are multiple lines of text boxes in the cell, the multiple lines of recognized text content are spliced ​​in order from top to bottom to obtain the complete content of the cell.

[0127] The method for recognizing tables in images provided by the embodiment is used to perform overall correction on images containing tables, which has good universality and reduces the loss of information in the image after correction; it can accurately recognize cost quota tables containing gray-white backgrounds or implicit cells separated by no background; it can fill the text area of ​​cells without text detection results, which can completely avoid the omission of single words in the overall text recognition results caused by missed detection by the text detection model; by merging and widening text boxes in the same row in the cell, it can provide space assistance for identifying spaces in subsequent text recognition, and avoid the partial or even complete omission of the first character or the last punctuation. After this processing, a nearly 100% accurate text detection area is provided for subsequent text recognition processing in the table.

[0128] Embodiment 3 Corresponding to the above-mentioned embodiment 1, embodiment 2 of the present invention provides a device for recognizing a table in an image. The corresponding technical feature details and corresponding technical effects can be referred to the above-mentioned embodiment 1, and will not be repeated in this embodiment. Fig.21 A block diagram of a device for recognizing a table in an image provided by Embodiment 3 of the present invention, such as Fig.21 As shown, the device includes: a detection module 601, an extraction module 602, a recognition module 603 and an output module 604.

[0129] The detection module 601 is used to detect cells in the image to be processed and obtain multiple cell coordinates; the extraction module 602 is used to extract the cell image in the image to be processed using the cell coordinates; the recognition module 603 is used to recognize the text in the cell image; and the output module 604 is used to construct a table in the image to be processed using the multiple cell coordinates and output the text in the cell image to the table.

[0130] Optionally, in one embodiment, the device further includes: a correction module, used to perform overall perspective correction on the image to be processed before the detection module detects cells in the image to be processed; the recognition module is also used to identify text outside the cell image after the extraction module extracts the cell image from the image to be processed using the cell coordinates; the output module is also used to output the text outside the cell image outside the table after constructing a table in the image to be processed using the multiple cell coordinates.

[0131] Optionally, in one embodiment, the correction module includes: a first processing unit, used to perform grayscale processing and binarization processing on the image to be processed in sequence to obtain a first image; a first identification unit, used to identify the quadrilateral with the largest area in the first image; an extension unit, used to extend the four sides of the quadrilateral with the largest area to the boundary of the first image; a translation unit, used to translate the four extended sides outward so that the extension line of each side can pass through at least one corner of the first image; a first calculation unit, used to calculate the four intersection points of the four sides after translation; and a correction unit, used to perform perspective correction on the part of the image to be processed defined by the four intersection points.

[0132] Optionally, in one embodiment, when the first recognition unit identifies the quadrilateral with the largest area in the first image, the specific steps performed include: Step S11: setting parameter values ​​of the contour search function and parameter values ​​of the polygon fitting function; Step S12: detecting the contour boundary line in the first image through the contour search function, performing polygon fitting on the contour boundary line in the first image through the polygon fitting function, and inputting the quadrilateral with the largest area into the quadrilateral set among the fitted polygons; Step S13: determining whether a preset loop end condition is met, if the loop end condition is not met, returning to step S11, if the loop end condition is met, executing step S14; Step S14: outputting the quadrilateral with the largest area in the first image according to the quadrilateral set.

[0133] Optionally, in one embodiment, the cells in the image to be processed include a first cell, and the detection module includes: a second recognition unit, used to identify the contour boundary line in the image to be processed; a first fitting unit, used to perform polygon fitting on the contour boundary line; a filtering unit, used to filter out quadrilaterals from the fitted polygons; a second fitting unit, used to perform rectangle fitting on the filtered quadrilaterals; and a second processing unit, used to obtain the coordinates of the first cell based on the fitted rectangle.

[0134] Optionally, in one embodiment, when the filtering unit filters out quadrilaterals from the fitted polygons, the specific steps performed include: determining the maximum area occupied by a single word based on the image resolution of the image to be processed and the font size in the table; and extracting quadrilaterals having an area larger than the maximum area occupied by the single word from the fitted polygons.

[0135] Optionally, in an embodiment, when the second fitting unit performs rectangular fitting on the filtered quadrilateral, the specific steps include: taking out four vertices of the quadrilateral, and sorting them in the order of upper left, upper right, lower right, and lower left according to the positions of the four vertices, and obtaining the coordinates of the four vertices as (X [左上] , Y [左上] ), (X [右上] , Y [右上] ), (X [右下] , Y [右下] ), (X [左下] , Y [左下] ); recalculate the coordinates of the upper left and lower right vertices as follows: X [左上] '= (X [左上] + X [左下] ) / 2, Y [左上] ' = (Y [左上] + X [右上] ) / 2, X [右下]' = (X [右下] + X [右上] ) / 2, Y [右下] '= (Y [右下] + Y [左下] ) / 2; recalculate the coordinates of the upper right vertex and the lower left vertex as follows: X [右上] '=X [右下] ', Y [右上] ' = Y [左上] ', X [左下] ' = X [左上] ', Y [左下] ' = Y [右下] ', where the four vertices of each quadrilateral (X [左上] ', Y [左上] '), (X [右上] ', Y [右上] '), (X [右下] ', Y [右下] '), (X [左下] ', Y [左下] ') form a rectangle.

[0136] Optionally, in one embodiment, when the second processing unit obtains the first cell coordinates based on the fitted rectangle, the specific steps performed include: generating a blank image based on the size of the image to be processed; drawing the fitted rectangle into the blank image to obtain a second image; performing edge detection on the second image and extracting line segments to obtain a line segment set; identifying horizontal and vertical lines in the line segment set to obtain a horizontal line set and a vertical line set; and determining the first cell coordinates based on the intersection points of the horizontal and vertical lines in the horizontal line set and the vertical line set.

[0137] Optionally, in one embodiment, the steps of identifying horizontal and vertical lines in the line segment set and obtaining the horizontal line set and the vertical line set include: judging whether the longitudinal coordinates of the two endpoints of the line segment are equal; if the longitudinal coordinates of the two endpoints of the line segment are equal, determining that the line segment is a horizontal line; judging whether the horizontal coordinates of the two endpoints of the line segment are equal; if the horizontal coordinates of the two endpoints of the line segment are equal, determining that the line segment is a vertical line; if both the horizontal and longitudinal coordinates of the two endpoints of the line segment are unequal, calculating the slope of the line segment; when the absolute value of the slope of the line segment is less than 1, determining that the line segment is a horizontal line, and when the absolute value of the slope of the line segment is greater than 1, determining that the line segment is a vertical line.

[0138] Optionally, in one embodiment, the step of determining the first cell coordinates according to the intersection points of the horizontal and vertical lines in the horizontal line set and the vertical line set includes: searching the horizontal lines in the horizontal line set for the horizontal lines in the same row to obtain a horizontal line row; traversing the horizontal line row to determine the relationship between adjacent horizontal lines in the row extension direction, wherein, if the current adjacent horizontal lines have overlapping parts or the gaps therein are less than a first preset number of pixels, the adjacent horizontal lines are merged into a new horizontal line, and the judgment is performed with the next adjacent horizontal line, and finally the horizontal line row is processed as a long horizontal line row; searching the vertical lines in the vertical line set for the vertical lines in the same column to obtain a vertical line column; traversing the vertical line column to determine the relationship between adjacent vertical lines in the column extension direction, wherein, if the current adjacent vertical lines have overlapping parts or the gaps therein are less than a second preset number of pixels, the adjacent vertical lines are merged into a new vertical line, and the judgment is performed with the next adjacent vertical line, and finally the vertical line column is processed as a long vertical line column; and determining the first cell coordinates according to the coordinates of the intersection points of the horizontal and vertical lines in the long horizontal line row and the long vertical line column.

[0139] Optionally, in one embodiment, the step of finding horizontal lines in the same row in the horizontal line set to obtain the horizontal line row includes: calculating the ordinate of the midpoint of each horizontal line in the horizontal line set; sorting the horizontal lines in the horizontal line set according to the ordinate of the midpoint; taking the adjacent horizontal lines in the sorted horizontal line set whose ordinates of the midpoints differ by a third preset number of pixels as the horizontal lines in the same row; traversing the horizontal line row to determine the relationship between adjacent horizontal lines in the row extension direction includes: step S21: sorting the horizontal lines in the horizontal line row according to the ordinate of the same end point of the line segment; step S22: determining the relationship between the first horizontal line and the second horizontal line, wherein the first horizontal line and the second horizontal line have a predetermined number of pixels. The initial value is the first horizontal line and the second horizontal line in the horizontal line row. When the first horizontal line and the second horizontal line are in a containing relationship, step S23 is executed. When the first horizontal line and the second horizontal line are in a partially overlapping relationship, or when there is a gap between the first horizontal line and the second horizontal line in the row extension direction, and the gap is less than or equal to the first preset number of pixels, step S24 is executed. Or when there is a gap between the first horizontal line and the second horizontal line in the row extension direction, and the gap is greater than the first preset number of pixels, step S26 is executed. Step S23: Determine that the longer horizontal line between the first horizontal line and the second horizontal line is the new horizontal line, and step S25 is executed. Step S24: taking the horizontal coordinates of the two most distant endpoints on the first horizontal line and the second horizontal line as the horizontal coordinates of the new horizontal line, taking the average of the vertical coordinates of the endpoints on the first horizontal line and the second horizontal line as the vertical coordinate of the new horizontal line, and executing step S25; Step S25: taking the new horizontal line as the first horizontal line, extracting the next horizontal line in the horizontal line row as the second horizontal line, and returning to step S22, wherein when all the horizontal lines in the horizontal line row are extracted, executing step S27; Step S26: storing the first horizontal line in the long horizontal line row, taking the second horizontal line as the first horizontal line, extracting the next horizontal line in the horizontal line row as the second horizontal line, and returning To step S22, wherein, when all horizontal lines in the horizontal line row are extracted, step S27 is executed; step S27: output the long horizontal line row; the step of finding the vertical lines in the same column in the vertical line set to obtain the vertical line column includes: calculating the horizontal coordinates of the midpoints of each vertical line in the vertical line set; sorting the vertical lines in the vertical line set into columns according to the horizontal coordinates of the midpoints; taking the adjacent vertical lines in the sorted vertical line set whose horizontal coordinates of the midpoints differ by a fourth preset number of pixels as the vertical lines in the same column; traversing the vertical line column, the step of determining the relationship between adjacent vertical lines in the column extension direction includes: step S31: sorting the vertical lines in the vertical line column into columns according to the vertical coordinates of the same end point of the line segment;Step S32: Determine the relationship between the first vertical line and the second vertical line, wherein the initial values ​​of the first vertical line and the second vertical line are the first vertical line and the second vertical line in the vertical line column, when the first vertical line and the second vertical line are in an inclusion relationship, execute step S33, when the first vertical line and the second vertical line are in a partially overlapping relationship, or when there is a gap between the first vertical line and the second vertical line in the column extension direction, and the gap is less than or equal to the second preset pixel number, execute step S34, or when there is a gap between the first vertical line and the second vertical line in the column extension direction, and the gap is greater than the second preset pixel number, execute step S36; Step S33: Determine that the longer vertical line between the first vertical line and the second vertical line is the new vertical line, and execute step S35; Step S34: The ordinates of the two most distant endpoints on the first vertical line and the second vertical line are used as the ordinates of the new vertical line, and the average of the abscissas of the endpoints on the first vertical line and the second vertical line is used as the abscissa of the new vertical line, and step S35 is executed; step S35: the new vertical line is used as the first vertical line, and the next vertical line in the vertical line column is extracted as the second vertical line, and the process returns to step S32, wherein when all the vertical lines in the vertical line column are extracted, step S37 is executed; step S36: the first vertical line is stored in the long vertical line column, the second vertical line is used as the first vertical line, and the next vertical line in the vertical line column is extracted as the second vertical line, and the process returns to step S32, wherein when all the vertical lines in the vertical line column are extracted, step S37 is executed; step S37: the long vertical line column is output. ;

[0140] Optionally, in one embodiment, the step of determining the coordinates of the first cell according to the coordinates of the intersections of the horizontal and vertical lines in the long horizontal line row and the long vertical line column includes: comparing the horizontal and vertical lines in the long horizontal line row and the long vertical line column in pairs to determine whether there is an intersection, obtaining a vertex when an intersection exists, and writing the vertex into a vertex set in sequence according to the row extension direction and the column extension direction; traversing the vertex set, performing the following steps: step S410: obtaining a first vertex, wherein the initial value of the first vertex is the first vertex in the vertex set; step S411: determining whether there is an adjacent next vertex to the first vertex in the row extension direction, if not, executing step S412, If it exists, execute step S413; step S412: take the next vertex in the vertex set as the first vertex, and return to step S410; step S413: take the next vertex adjacent to the first vertex in the row extension direction as the second vertex, and take the first vertex as the first vertex of the first cell; step S414: determine whether the second vertex has an adjacent next vertex in the column extension direction, if not, execute step S415, if yes, execute step S417; step S415: determine whether the second vertex has an adjacent next vertex in the row extension direction, if not, execute step S412, if yes, execute step S416; step S416: The next vertex adjacent to the second vertex in the column extension direction is taken as the second vertex, and the process returns to step S414; step S417: the next vertex adjacent to the second vertex in the column extension direction is taken as the third vertex, and the second vertex is taken as the second vertex of the first cell; step S418: determine whether the third vertex has an adjacent previous vertex in the row extension direction, if not, execute step S419, if yes, execute step S421; step S419: determine whether the third vertex has an adjacent next vertex in the column extension direction, if not, execute step S412, if yes, execute step S420; step S420: the next vertex adjacent to the third vertex in the column extension direction is taken as is the third vertex, and returns to step S418; step S421: taking the previous vertex adjacent to the third vertex in the row extension direction as the fourth vertex, and taking the third vertex as the third vertex of the first cell; step S422: judging whether there is an adjacent previous vertex of the fourth vertex in the column extension direction, if not, executing step S423, if yes, executing step S425; step S423: judging whether there is an adjacent previous vertex of the fourth vertex in the row extension direction, if not, executing step S412, if yes, executing step S424; step S424: taking the previous vertex adjacent to the fourth vertex in the row extension direction as the fourth vertex, and returning to step S422;Step S425: take the previous vertex adjacent to the fourth vertex in the column extension direction as the fifth vertex, and take the fourth vertex as the fourth vertex of the first cell; Step S426: determine whether the fifth vertex is the same vertex as the first vertex, if not, execute step S427, if yes, execute step S429; Step S427: determine whether the fifth vertex has an adjacent previous vertex in the column extension direction, if not, execute step S412, if yes, execute step S428; Step S428: take the previous vertex adjacent to the fifth vertex in the column extension direction as the fifth vertex, and return to step S426; Step S429: determine the coordinates of the first cell using the coordinates of the first vertex of the first cell, the second vertex of the first cell, the third vertex of the first cell and the fourth vertex of the first cell. ;

[0141] Optionally, in one embodiment, the image to be processed includes rows that are alternately distinguished by a first background color and a second background color, and the step of identifying contour boundary lines in the image to be processed includes: converting the foreground text in the image to be processed into black, converting the first background color into black, and converting the second background color into white to obtain a first converted image; converting the foreground text in the image to be processed into black and converting the entire background into white to obtain a second converted image; detecting contour boundary lines in the first converted image and the second converted image, wherein polygon fitting is performed on the contour boundary lines in the first converted image and the second converted image, respectively, quadrilaterals are filtered out from the fitted polygons, and rectangle fitting is performed on the filtered quadrilaterals to obtain a group of rectangle sets; the step of drawing the fitted rectangles into the blank image includes: drawing both groups of rectangle sets into the blank image.

[0142] Optionally, in one embodiment, the first background color is gray, the second background color is white, the foreground text in the image to be processed is converted to black, the first background color is converted to black, and the second background color is converted to white, and the step of obtaining a first converted image includes: after grayscale processing of the image to be processed, the pixel values ​​of all points in the image to be processed with pixel values ​​less than 240 are set to 0, and the pixel values ​​of all points greater than or equal to 240 are set to 255; the foreground text in the image to be processed is converted to black, and the entire background is converted to white, and the step of obtaining a second converted image includes: after grayscale processing of the image to be processed, the minimum pixel value of the points in the first background color area is found, and the pixel values ​​of all points in the image to be processed with pixel values ​​greater than the minimum pixel value are set to 255.

[0143] Optionally, in one embodiment, the cells of the image to be processed also include a second cell located within the first cell and without a dividing marker. When the second recognition unit identifies the contour boundary line in the image to be processed, the specific steps performed include: performing grayscale processing on the image to be processed to obtain a grayscale image; detecting the contour boundary line in the grayscale image; the step of detecting the cells in the image to be processed also includes: identifying a preset character area in the first cell according to text features of preset characters; calculating the midline of each of the preset character areas in the first cell to obtain a midline set corresponding to the first cell; and determining the second cell according to the midline set.

[0144] Optionally, in one embodiment, the step of determining the second cell according to the centerline set includes: step S50: extracting a first edge line, wherein the initial value of the first edge line is the first edge line of the first cell in the row extension direction; step S51: determining a center line adjacent to the first edge line in the centerline set to obtain a reference center line; step S52: determining whether the reference center line is the last center line in the centerline set, if not, executing step S53, if yes, executing step S56; step S53: calculating the distance between the first edge line and the reference center line to obtain a half-line spacing, translating the reference center line in a direction away from the second edge line by the half-line spacing to obtain a second edge line; step S54: determining that the first edge line, the second edge line, and a portion of the first cell intercepted by the first edge line and the second edge line are the second cell; step S55: taking the second edge line as the first edge line, and returning to step S50; step S56: determining that the first edge line, the second edge line of the first cell in the row extension direction, and a portion of the first cell intercepted by the first edge line and the second edge line are the second cell.

[0145] Optionally, in one embodiment, the step of identifying a preset character area in the first cell according to text features of preset characters includes: detecting the text area in the image to be processed to obtain a first text area; cross-judging the first text area and the first cell to determine the first text area in the first cell; and for the first text area in the first cell, filtering out the first text area containing the preset character according to the text features of the preset character to obtain the preset character area in the first cell.

[0146] Optionally, in one embodiment, the recognition module includes: a detection unit, used to detect the second text area in the cell image; a first determination unit, used to determine the second text area located in the same row among the multiple detected second text areas to obtain a text area row; a second determination unit, used to determine the upper and lower edges of the text area row extending in the row extension direction based on the vertices of the second text area; a third determination unit, used to determine the left and right edges of the text area row based on the boundary line of the cell in the column extension direction; a fourth determination unit, used to determine the target area using the upper, lower, left and right edges of the text area row; and a recognition unit, used to recognize the text of the target area to obtain the text in the cell image.

[0147] Optionally, in one embodiment, the recognition module also includes: an acquisition unit, used to acquire preset parameters corresponding to the typeset features of the table when the detection unit cannot detect the second text area; a fifth determination unit, used to determine the upper and lower borders of the suspected text area in the cell image according to the preset parameters; a sixth determination unit, used to determine the left and right borders of the suspected text area based on the boundary line of the cell in the column extension direction; and a seventh determination unit, used to determine the target area using the upper, lower, left and right borders of the suspected text area row.

[0148] Embodiment 4 This embodiment also provides a computer device, such as a smart phone, tablet computer, laptop computer, desktop computer, rack server, blade server, tower server or cabinet server (including an independent server or a server cluster composed of multiple servers) that can execute programs. Fig. 22 As shown, the computer device 01 of this embodiment includes at least but is not limited to: a memory 011 and a processor 012 which can be interconnected through a system bus. Fig. 22 It should be pointed out that Fig. 22 Only a computer device 01 having components memory 011 and processor 012 is shown, but it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0149] In this embodiment, the memory 011 (i.e., readable storage medium) includes flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 011 may be an internal storage unit of the computer device 01, such as a hard disk or memory of the computer device 01. In other embodiments, the memory 011 may also be an external storage device of the computer device 01, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card, etc. equipped on the computer device 01. Of course, the memory 011 may also include both the internal storage unit of the computer device 01 and its external storage device. In this embodiment, the memory 011 is generally used to store the operating system and various application software installed on the computer device 01, such as the program code of the image table recognition device in the third embodiment. In addition, the memory 011 can also be used to temporarily store various types of data that have been output or are to be output.

[0150] In some embodiments, the processor 012 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips. The processor 012 is generally used to control the overall operation of the computer device 01. In this embodiment, the processor 012 is used to run the program code stored in the memory 011 or process data, such as a method for recognizing a table in an image.

[0151] Embodiment 5 This embodiment also provides a computer-readable storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory (for example, an SD or DX memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a disk, an optical disk, a server, an App application store, etc., on which a computer program is stored, and the program realizes corresponding functions when executed by a processor. The computer-readable storage medium of this embodiment is used to store a device for recognizing a table in an image, and when executed by a processor, the method for recognizing a table in an image of embodiment 1 is realized.

[0152] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0153] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0154] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method.

[0155] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A method for recognizing a table in an image, characterized in that: include: Detect cells in the image to be processed and obtain multiple cell coordinates; Extracting a cell image from the image to be processed using the cell coordinates; Recognizing text in the cell image; constructing a table in the image to be processed using the multiple cell coordinates; as well as The texts in each of the cell images are outputted correspondingly into the table.

2. The method for recognizing a table in an image according to claim 1, characterized in that: The method further comprises: Before the step of detecting cells in the image to be processed, performing overall perspective correction on the image to be processed; After the step of extracting a cell image from the image to be processed using the cell coordinates, identifying text outside the cell image; After the step of constructing a table in the image to be processed using the plurality of cell coordinates, the text outside the cell image is output outside the table.

3. The method for recognizing a table in an image according to claim 2, characterized in that: The step of performing overall perspective correction on the image to be processed comprises: Performing grayscale processing and binarization processing on the image to be processed in sequence to obtain a first image; Identifying the quadrilateral with the largest area in the first image; Extending four sides of the quadrilateral with the largest area to the boundary of the first image; translating the four extended sides outward so that the extended line of each side can at least pass through a corner of the first image; Calculating four intersection points of the four translated edges; and A perspective correction is performed on a portion of the image to be processed that is defined by the four intersection points.

4. The method for recognizing a table in an image according to claim 3, characterized in that: The step of identifying the quadrilateral with the largest area in the first image comprises: Step S11: setting parameter values ​​of the contour search function and parameter values ​​of the polygon fitting function; Step S12: detecting the contour boundary line in the first image by using the contour search function, performing polygon fitting on the contour boundary line in the first image by using the polygon fitting function, and inputting the quadrilateral with the largest area among the fitted polygons into the quadrilateral set; Step S13: Determine whether a preset cycle end condition is met. If the cycle end condition is not met, return to step S11. If the cycle end condition is met, execute step S14. Step S14: outputting the quadrilateral with the largest area in the first image according to the quadrilateral set.

5. The method for recognizing a table in an image according to claim 1, characterized in that: The cell in the image to be processed includes a first cell, and the step of detecting the cell in the image to be processed includes: Identifying a contour boundary line in the image to be processed; Performing polygon fitting on the contour boundary line; Filter out quadrilaterals from the fitted polygons; Perform rectangle fitting on the filtered quadrilaterals; The first cell coordinates are obtained according to the fitted rectangle.

6. The method for recognizing a table in an image according to claim 5, characterized in that: The steps of filtering out quadrilaterals from the fitted polygons include: Determining the maximum area occupied by a single word according to the image resolution of the image to be processed and the font size in the table; and A quadrilateral having an area larger than the maximum area occupied by the single word is extracted from the fitted polygons.

7. The method for recognizing a table in an image according to claim 5, characterized in that: The steps of performing rectangular fitting on the filtered quadrilaterals include: Take out the four vertices of the quadrilateral, and sort them in the order of upper left, upper right, lower right, and lower left according to the positions of the four vertices, and obtain the coordinates of the four vertices as (X [左上] , Y [左上] ), (X [右上] , Y [右上] ), (X [右下] , Y [右下] ), (X [左下] , Y [左下] ); Recalculate the coordinates of the upper left and lower right vertices as follows: X [左上] ' = (X [左上] + X [左下] ) / 2, AND [左上] ' = (And [左上] + X [右上] ) / 2, X [右下] ' = (X [右下] + X [右上] ) / 2, AND [右下] ' = (And [右下] + And [左下] ) / 2; Recalculate the coordinates of the upper right and lower left vertices as follows: X [右上] '= X [右下] ' AND [右上] '= And [左上] ' X [左下] ' = X [左上] ' AND [左下] '= And [右下] ', Among them, the four vertices (X [左上] ', Y [左上] '), (X [右上] ', Y [右上] '), (X [右下] ', Y [右下] '), (X [左下] ', Y [左下] ') form a rectangle.

8. The method for recognizing a table in an image according to claim 5, characterized in that: The step of obtaining the coordinates of the first cell according to the fitted rectangle includes: Generate a blank image according to the size of the image to be processed; Drawing the fitted rectangle into the blank image to obtain a second image; Performing edge detection on the second image and extracting line segments to obtain a line segment set; Identifying horizontal lines and vertical lines in the line segment set to obtain a horizontal line set and a vertical line set; and The first cell coordinates are determined according to intersections of horizontal lines and vertical lines in the horizontal line set and the vertical line set.

9. The method for recognizing a table in an image according to claim 8, characterized in that: The step of identifying the horizontal lines and vertical lines in the line segment set to obtain the horizontal line set and the vertical line set comprises: Determine whether the ordinates of the two end points of the line segment are equal; If the ordinates of the two end points of the line segment are equal, the line segment is determined to be a horizontal line; Determine whether the horizontal coordinates of the two end points of the line segment are equal; If the horizontal coordinates of the two end points of the line segment are equal, the line segment is determined to be a vertical line; If the horizontal coordinates and vertical coordinates of the two end points of the line segment are not equal, calculate the slope of the line segment; When the absolute value of the slope of the line segment is less than 1, the line segment is determined to be a horizontal line, and when the absolute value of the slope of the line segment is greater than 1, the line segment is determined to be a vertical line.

10. The method for recognizing a table in an image according to claim 8, characterized in that: The step of determining the first cell coordinates according to the intersection points of the horizontal lines and the vertical lines in the horizontal line set and the vertical line set comprises: Find the horizontal lines in the same row in the horizontal line set to obtain a horizontal line row; Traversing the horizontal line row, determining the relationship between adjacent horizontal lines in the row extension direction, wherein if the current adjacent horizontal lines have overlapping parts or the gap between them is less than a first preset number of pixels, merging the adjacent horizontal lines into a new horizontal line, determining the relationship between the adjacent horizontal lines and the next adjacent horizontal line, and finally processing the horizontal line row as a long horizontal line row; Find the vertical lines in the same column in the vertical line set to obtain a vertical line column; Traversing the vertical line column, determining the relationship between adjacent vertical lines in the column extension direction, wherein if the current adjacent vertical lines have overlapping parts or the gap between them is less than a second preset number of pixels, merging the adjacent vertical lines into a new vertical line, determining the relationship between the adjacent vertical lines and the next adjacent vertical line, and finally processing the vertical line column into a long vertical line column; and The first cell coordinates are determined according to the coordinates of the intersections of the horizontal and vertical lines in the long horizontal line row and the long vertical line column.

11. The method for recognizing a table in an image according to claim 10, characterized in that: The step of searching for horizontal lines in the same row in the horizontal line set to obtain the horizontal line row includes: calculating the ordinate of the midpoint of each horizontal line in the horizontal line set; sorting the horizontal lines in the horizontal line set according to the ordinate of the midpoint; and treating the adjacent horizontal lines in the sorted horizontal line set whose ordinates of the midpoints differ by a third preset number of pixels as the horizontal lines in the same row; The step of traversing the horizontal line row and determining the relationship between adjacent horizontal lines in the row extension direction comprises: Step S21: sorting the horizontal lines in the horizontal line row according to the horizontal coordinates of the same endpoints of the line segments; Step S22: determining the relationship between the first horizontal line and the second horizontal line, wherein the initial values ​​of the first horizontal line and the second horizontal line are the first horizontal line and the second horizontal line in the horizontal line row, when the first horizontal line and the second horizontal line are in a containing relationship, executing step S23, when the first horizontal line and the second horizontal line are in a partially overlapping relationship, or when there is a gap between the first horizontal line and the second horizontal line in the row extension direction, and the gap is less than or equal to the first preset pixel number, executing step S24, or when there is a gap between the first horizontal line and the second horizontal line in the row extension direction, and the gap is greater than the first preset pixel number, executing step S26; Step S23: determining that the longer horizontal line between the first horizontal line and the second horizontal line is the new horizontal line, and executing step S25; Step S24: taking the abscissas of the two most distant endpoints on the first horizontal line and the second horizontal line as the abscissas of the new horizontal line, taking the average of the ordinates of the endpoints on the first horizontal line and the second horizontal line as the ordinate of the new horizontal line, and executing step S25; Step S25: taking the new horizontal line as the first horizontal line, extracting the next horizontal line in the horizontal line row as the second horizontal line, and returning to step S22, wherein when all horizontal lines in the horizontal line row are extracted, executing step S27; Step S26: storing the first horizontal line in the long horizontal line row, taking the second horizontal line as the first horizontal line, extracting the next horizontal line in the horizontal line row as the second horizontal line, and returning to step S22, wherein when all horizontal lines in the horizontal line row are extracted, executing step S27; Step S27: outputting the long horizontal line row; The step of searching for vertical lines in the same column in the vertical line set to obtain the vertical line column includes: calculating the horizontal coordinates of the midpoints of each vertical line in the vertical line set; sorting the vertical lines in the vertical line set into columns according to the horizontal coordinates of the midpoints; and treating the adjacent vertical lines in the sorted vertical line set whose horizontal coordinates of the midpoints differ by a fourth preset number of pixels as vertical lines in the same column; The step of traversing the vertical line column and determining the relationship between adjacent vertical lines in the column extension direction comprises: Step S31: sorting the vertical lines in the vertical line column according to the ordinates of the same end points of the line segments; Step S32: determining the relationship between the first vertical line and the second vertical line, wherein the initial values ​​of the first vertical line and the second vertical line are the first vertical line and the second vertical line in the vertical line column, when the first vertical line and the second vertical line are in an inclusive relationship, executing step S33, when the first vertical line and the second vertical line are in a partially overlapping relationship, or when there is a gap between the first vertical line and the second vertical line in the column extension direction, and the gap is less than or equal to the second preset pixel number, executing step S34, or when there is a gap between the first vertical line and the second vertical line in the column extension direction, and the gap is greater than the second preset pixel number, executing step S36; Step S33: determining that the longer vertical line between the first vertical line and the second vertical line is the new vertical line, and executing step S35; Step S34: taking the ordinates of the two most distant endpoints on the first vertical line and the second vertical line as the ordinate of the new vertical line, taking the average of the abscissas of the endpoints on the first vertical line and the second vertical line as the abscissa of the new vertical line, and executing step S35; Step S35: taking the new vertical line as the first vertical line, extracting the next vertical line in the vertical line column as the second vertical line, and returning to step S32, wherein when all vertical lines in the vertical line column are extracted, executing step S37; Step S36: storing the first vertical line in the long vertical line column, taking the second vertical line as the first vertical line, extracting the next vertical line in the vertical line column as the second vertical line, and returning to step S32, wherein when all vertical lines in the vertical line column are extracted, executing step S37; Step S37: Output the long vertical line column.

12. The method for recognizing a table in an image according to claim 10, characterized in that: The step of determining the first cell coordinates according to the coordinates of the intersections of the horizontal and vertical lines in the long horizontal line row and the long vertical line column comprises: Compare the horizontal lines and vertical lines in the long horizontal line row and the long vertical line column in pairs to determine whether there is an intersection, obtain a vertex when there is an intersection, and write the vertex into a vertex set in sequence according to the row extension direction and the column extension direction; Traverse the vertex set and perform the following steps: Step S410: Obtain a first vertex, wherein an initial value of the first vertex is the first vertex in the vertex set; Step S411: Determine whether there is an adjacent next vertex to the first vertex in the row extension direction, if not, execute step S412, if yes, execute step S413; Step S412: taking the next vertex in the vertex set as the first vertex, and returning to step S410; Step S413: taking the next vertex adjacent to the first vertex in the row extension direction as the second vertex, and taking the first vertex as the first vertex of the first cell; Step S414: Determine whether the second vertex has an adjacent next vertex in the column extension direction, if not, execute step S415, if yes, execute step S417; Step S415: Determine whether there is an adjacent next vertex to the second vertex in the row extension direction, if not, execute step S412, if yes, execute step S416; Step S416: taking the next vertex adjacent to the second vertex in the row extension direction as the second vertex, and returning to step S414; Step S417: taking the next vertex adjacent to the second vertex in the column extension direction as the third vertex, and taking the second vertex as the second vertex of the first cell; Step S418: Determine whether the third vertex has an adjacent previous vertex in the row extension direction, if not, execute step S419, if yes, execute step S421; Step S419: Determine whether the third vertex has an adjacent next vertex in the column extension direction. If not, execute step S412; if so, execute step S420; Step S420: taking the next vertex adjacent to the third vertex in the column extension direction as the third vertex, and returning to step S418; Step S421: taking the previous vertex adjacent to the third vertex in the row extension direction as the fourth vertex, and taking the third vertex as the third vertex of the first cell; Step S422: Determine whether the fourth vertex has an adjacent previous vertex in the column extension direction, if not, execute step S423, if yes, execute step S425; Step S423: Determine whether the fourth vertex has an adjacent previous vertex in the row extension direction, if not, execute step S412, if yes, execute step S424; Step S424: taking the previous vertex adjacent to the fourth vertex in the row extension direction as the fourth vertex, and returning to step S422; Step S425: taking the previous vertex adjacent to the fourth vertex in the column extension direction as the fifth vertex, and taking the fourth vertex as the fourth vertex of the first cell; Step S426: Determine whether the fifth vertex is the same vertex as the first vertex, if not, execute step S427, if yes, execute step S429; Step S427: Determine whether the fifth vertex has an adjacent previous vertex in the column extension direction, if not, execute step S412, if yes, execute step S428; Step S428: taking the previous vertex adjacent to the fifth vertex in the column extension direction as the fifth vertex, and returning to step S426; Step S429: Determine the coordinates of the first cell using the coordinates of the four vertices, namely, the first vertex of the first cell, the second vertex of the first cell, the third vertex of the first cell and the fourth vertex of the first cell.

13. The method for recognizing a table in an image according to claim 8, characterized in that: The image to be processed includes rows that are alternately distinguished by a first background color and a second background color, and the step of identifying a contour boundary line in the image to be processed includes: Converting the foreground text in the image to be processed into black, converting the first background color into black, and converting the second background color into white to obtain a first converted image; Converting the foreground text in the image to be processed into black and converting all the background into white to obtain a second converted image; Detecting contour boundary lines in the first conversion image and the second conversion image, wherein polygon fitting is performed on the contour boundary lines in the first conversion image and the second conversion image, quadrilaterals are filtered out from the fitted polygons, and rectangle fitting is performed on the filtered quadrilaterals to obtain a set of rectangle sets; The step of drawing the fitted rectangle into the blank image includes: drawing both sets of the rectangle sets into the blank image.

14. The method for recognizing a table in an image according to claim 13, characterized in that: The first background color is gray, the second background color is white, The step of converting the foreground text in the image to be processed into black, converting the first background color into black, and converting the second background color into white to obtain a first converted image comprises: after grayscale processing of the image to be processed, setting all pixel values ​​of points with pixel values ​​less than 240 in the image to be processed to 0, and setting all pixel values ​​of points with pixel values ​​greater than or equal to 240 to 255; The foreground text in the image to be processed is converted into black, and the entire background is converted into white, and the step of obtaining a second converted image includes: performing grayscale processing on the image to be processed, finding the minimum pixel value of the points in the first background color area, and setting the pixel values ​​of all points in the image to be processed whose pixel values ​​are greater than the minimum pixel value to 255.

15. The method for recognizing a table in an image according to claim 8, characterized in that: The cells of the image to be processed also include a second cell located in the first cell and having no boundary marker, The step of identifying the contour boundary line in the image to be processed includes: performing grayscale processing on the image to be processed to obtain a grayscale image; detecting the contour boundary line in the grayscale image; The step of detecting cells in the image to be processed also includes: Identifying a preset character region in the first cell according to text features of the preset characters; Calculating the center line of each of the preset character areas in the first cell to obtain a center line set corresponding to the first cell; The second cell is determined according to the center line set.

16. The method for recognizing a table in an image according to claim 15, characterized in that: The step of determining the second cell according to the midline set comprises: Step S50: extracting a first edge line, wherein an initial value of the first edge line is a first edge line of the first cell in a row extension direction; Step S51: determining a center line adjacent to the first side line in the center line set to obtain a reference center line; Step S52: determine whether the reference center line is the last center line in the center line set, if not, execute step S53, if yes, execute step S56; Step S53: calculating the distance between the first sideline and the reference centerline to obtain a half-line distance, and translating the reference centerline in a direction away from the second sideline by the half-line distance to obtain a second sideline; Step S54: determining the first edge line, the second edge line, and the portion of the first cell intercepted by the first edge line and the second edge line as the second cell; Step S55: taking the second edge line as the first edge line, and returning to step S50; Step S56: Determine the first sideline, the second sideline of the first cell in the row extension direction, and the portion of the first cell intercepted by the first sideline and the second sideline as the second cell.

17. The method for recognizing a table in an image according to claim 15, characterized in that: The step of identifying the preset character area in the first cell according to the text features of the preset characters comprises: Detecting a text region in the image to be processed to obtain a first text region; Performing a cross judgment on the first text area and the first cell to determine the first text area in the first cell; and For the first text area in the first cell, the first text area containing the preset character is screened out according to the text feature of the preset character to obtain the preset character area in the first cell.

18. The method for recognizing a table in an image according to claim 1, characterized in that: The step of recognizing text in the cell image comprises: Detecting a second text area in the cell image; Determine second text areas located in the same row among the multiple detected second text areas to obtain a text area row; Determine the upper and lower edges of the text area row extending in the row extension direction according to the vertices of the second text area; Determine the left and right borders of the text area row according to the boundary lines of the cell in the column extension direction; Determine the target area using the top, bottom, left and right borders of the text area line; The text in the target area is identified to obtain the text in the cell image.

19. The method for recognizing a table in an image according to claim 18, characterized in that: The step of identifying the text in the cell image also includes: When the second text area cannot be detected, obtaining preset parameters corresponding to the typesetting features of the table; Determine the upper and lower borders of the suspected text area in the cell image according to the preset parameters; Determine the left and right borders of the suspected text area according to the boundary lines of the cell in the column extension direction; The target area is determined using the top, bottom, left and right borders of the suspected text area line.

20. A device for recognizing a table in an image, characterized in that: include: A detection module, used for detecting cells in the image to be processed and obtaining multiple cell coordinates; An extraction module, used for extracting a cell image from the image to be processed using the cell coordinates; A recognition module, used for recognizing text in the cell image; as well as An output module is used to construct a table in the image to be processed using the multiple cell coordinates, and output the text in the cell image to the table.

21. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 19 are implemented.

22. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 19 are implemented.

Images