A method, apparatus, device, and readable storage medium for table recognition.

Abstract

This invention relates to the field of computer vision technology and discloses a method, apparatus, device, and readable storage medium for table recognition. The method includes: acquiring an image to be recognized; inputting the image to be recognized into a preset table detection model to identify a table image from the image; inputting the table image into the preset table recognition model to determine the endpoint coordinates of each cell in the table image and the connection relationships between the endpoints; and generating a target table based on the endpoint coordinates and connection relationships. By implementing this invention, table recognition is achieved through endpoint extraction and endpoint connection relationship extraction, eliminating reliance on table line extraction, ensuring the accuracy of table structure recognition, and improving the accuracy of table information recognition.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and specifically to a table recognition method, apparatus, device, and readable storage medium. Background Technology

[0002] With the development of computer technology, text recognition has been widely applied in various fields, and the structured recognition of tables in images is one of the key tasks in text recognition. Existing table recognition methods usually extract lines from images based on computer vision algorithms or deep neural network models, and then use the extracted lines to draw tables to achieve table recognition in images.

[0003] However, in practical applications, table recognition algorithms face various interferences, such as image rotation due to shooting issues, partial table occlusion, incomplete table capture, and table distortion caused by paper wrinkles. Current methods for table recognition, which extract table lines to draw the table, may suffer from discontinuous lines or detection errors if affected by factors such as table image quality. This can lead to incorrect table structure and low recognition accuracy. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide a table recognition method, apparatus, device, and readable storage medium to solve the problem of low accuracy in table information recognition.

[0005] According to a first aspect, embodiments of the present invention provide a table recognition method, comprising: acquiring an image to be recognized; inputting the image to be recognized into a preset table detection model to recognize a table image from the image to be recognized; inputting the table image into the preset table recognition model to determine the endpoint coordinates of each cell in the table image and the connection relationship between each endpoint; and generating a target table based on the endpoint coordinates and the connection relationship.

[0006] The table recognition method provided in this invention determines the position of each cell by identifying the endpoint coordinates of each cell in a table image and the connection relationships between adjacent endpoints, thereby identifying the target table in the image. This method does not rely on table line extraction; it achieves table recognition through endpoint extraction and endpoint connection relationship extraction, ensuring the accuracy of table structure recognition and improving the accuracy of table information recognition.

[0007] In conjunction with the first aspect, in the first embodiment of the first aspect, the step of inputting the image to be identified into a preset table detection model and identifying a table image from the image to be identified includes: identifying the image to be identified based on the preset table detection model, obtaining a table region and the border endpoints corresponding to the table region; determining whether the table region satisfies table attribute conditions based on the border endpoints; when the table region does not satisfy the table attribute conditions, correcting the table region based on the border endpoints to obtain the table image.

[0008] The table recognition method provided in this invention detects whether the border endpoints of the table region meet the table attribute conditions. When a table region that does not meet the table attribute conditions is detected, it is corrected to obtain a table region that meets the table attribute conditions. This overcomes the problem of inaccurate table structure recognition caused by table image rotation or distortion, and ensures the recognition accuracy of table images to the greatest extent.

[0009] In conjunction with the first embodiment of the first aspect, in the second embodiment of the first aspect, the step of correcting the table region based on the border endpoints to obtain the table image includes: determining the minimum bounding rectangle corresponding to the table region based on the border endpoints; performing a perspective transformation based on the width and height of the minimum bounding rectangle to obtain a perspective transformation matrix from the border endpoints to the perspective transformation coordinates; and correcting the table region in the image to be identified based on the perspective transformation matrix and the minimum bounding rectangle to obtain the table image.

[0010] The table recognition method provided in this embodiment of the invention corrects the outer frame of the table area by using the border endpoints to obtain a corrected table image, thereby achieving accurate recognition of the table outer frame.

[0011] In conjunction with the first aspect, in the third embodiment of the first aspect, the step of inputting the table image into a preset table recognition model to determine the endpoint coordinates of each cell in the table image and the connection relationship between each endpoint includes: extracting endpoint region feature maps and endpoint connection relationship feature maps of the table image based on the preset table recognition model; performing a pooling operation on the endpoint region feature maps to obtain endpoint region pooled feature maps; extracting the endpoint coordinates of each cell from the endpoint region pooled feature maps; performing binarization processing on the connection relationship feature maps to obtain binarized connection relationship feature maps; and extracting the connection relationship between each endpoint from the binarized connection relationship feature maps.

[0012] The table recognition method provided in this invention extracts the endpoint coordinates of each cell and the connection relationship between each endpoint, so as to generate each cell according to the connection relationship between each endpoint. Therefore, it does not rely on table lines and avoids the problem of inaccurate table recognition caused by line distortion or discontinuity.

[0013] In conjunction with the third embodiment of the first aspect, in the fourth embodiment of the first aspect, the step of extracting the endpoint coordinates of each cell from the endpoint region pooling feature map includes: obtaining the first pixel value of each endpoint region in the endpoint region feature map and the second pixel value of each endpoint region in the endpoint region pooling feature map; comparing the first pixel value and the second pixel value from the first pixel to determine a plurality of target pixels whose first pixel value is the same as the second pixel value and is greater than a preset pixel value; and sorting the coordinates of the plurality of target pixels to obtain the endpoint coordinates of each cell.

[0014] The table recognition method provided in this invention determines the target pixel point as the endpoint of the cell by detecting pixel value, then sorts the pixels according to their coordinates to obtain the endpoint coordinates of each row cell and each column cell, and then restores the cell according to the endpoint coordinates, thereby improving the recognition accuracy of the cell.

[0015] In conjunction with the fourth embodiment of the first aspect, in the fifth embodiment of the first aspect, sorting the coordinates of the plurality of target pixels to obtain the endpoint coordinates of each cell includes: arranging the coordinates of the plurality of target pixels in rows according to a preset row sorting method to obtain a row sorting result; traversing the ordinates of each target pixel in the row sorting result to determine whether the difference between the ordinates of two adjacent target pixels is less than a first preset threshold; when the difference between the ordinates of two adjacent target pixels is less than the first preset threshold, determining that the two adjacent target pixels are in the same row; and determining the endpoint coordinates of the starting cell and the ending cell of each row to obtain the endpoint coordinates of each cell sorted by row.

[0016] The table recognition method provided in this invention arranges the coordinates of multiple target pixels by a preset row sorting method in order to determine the endpoint coordinates of each cell in the same row, thereby improving the accuracy of cell row recognition.

[0017] In conjunction with the fourth or fifth embodiment of the first aspect, in the sixth embodiment of the first aspect, sorting the coordinates of the plurality of target pixels to obtain the endpoint coordinates of each cell includes: arranging the coordinates of the plurality of target pixels in columns according to a preset column sorting method to obtain a column sorting result; traversing the horizontal coordinates of each target pixel in the column sorting result to determine whether the difference between the horizontal coordinates of two adjacent target pixels is less than a second preset threshold; when the difference between the vertical coordinates of two adjacent target pixels is less than the second preset threshold, determining that the two adjacent target pixels are in the same column; and determining the endpoint coordinates of the starting cell and the ending cell of each column to obtain the endpoint coordinates of each cell sorted by column.

[0018] The table recognition method provided in this embodiment of the invention arranges the coordinates of multiple target pixels by a preset column sorting method in order to determine the endpoint coordinates of each cell in the same column, thereby improving the column recognition accuracy of the cells.

[0019] In conjunction with the third embodiment of the first aspect, in the seventh embodiment of the first aspect, the step of extracting the connection relationship between each endpoint from the binarized connection relationship feature map includes: obtaining the coordinates of adjacent endpoints in the binarized connection relationship feature map; determining the set of connecting pixels between adjacent endpoints based on the coordinates of the adjacent endpoints; determining that there is a connection relationship between the adjacent endpoints when the average pixel value corresponding to the set of connecting pixels is greater than a third preset threshold; wherein, the connection relationship includes horizontal connection relationship and vertical connection relationship.

[0020] The table recognition method provided in this invention determines whether there is a connection between adjacent endpoints by detecting the average pixel value of the pixel set of connecting pixels between adjacent endpoints. This enables the determination of the corresponding horizontal and vertical connection relationships based on the endpoint coordinates, thus realizing the connection between adjacent endpoints and achieving accurate cell recognition.

[0021] In conjunction with the first aspect, in the eighth embodiment of the first aspect, the method further includes: determining the target position of each cell in the target table; detecting the text information in each cell to obtain text information; and generating target table information based on the target position and the text information.

[0022] The table recognition method provided in this invention identifies the corresponding text information by the target position of each cell, and determines the target table information based on the target position and the text information, thereby achieving the matching of cells and their text information.

[0023] According to a second aspect, embodiments of the present invention provide a table recognition device, comprising: an acquisition module for acquiring an image to be recognized; a table detection module for inputting the image to be recognized into a preset table detection model to recognize a table image from the image to be recognized; a table recognition module for inputting the table image into the preset table recognition model to determine the endpoint coordinates of each cell in the table image and the connection relationship between each endpoint; a determination module for determining the target position of each cell in the table image based on the endpoint coordinates and the connection relationship; a text detection module for detecting text information in each cell to obtain text information; and a generation module for generating target table information based on the target position and the text information.

[0024] According to a third aspect, an embodiment of the present invention provides an electronic device, including: a memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the computer instructions to perform the table recognition method described in the first aspect or any embodiment of the first aspect.

[0025] According to a fourth aspect, embodiments of the present invention provide a computer-readable storage medium storing computer instructions for causing a computer to perform the table recognition method described in the first aspect or any embodiment of the first aspect.

[0026] It should be noted that the beneficial effects of the table recognition device, electronic device, and computer-readable storage medium provided in the embodiments of the present invention can be found in the description of the corresponding content in the table recognition method, and will not be repeated here. Attached Figure Description

[0027] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0028] Figure 1 This is a flowchart of a table recognition method according to an embodiment of the present invention;

[0029] Figure 2 This is another flowchart of a table recognition method according to an embodiment of the present invention;

[0030] Figure 3 This is yet another flowchart of a table recognition method according to an embodiment of the present invention;

[0031] Figure 4 This is a structural block diagram of a table recognition device according to an embodiment of the present invention;

[0032] Figure 5 This is a schematic diagram of the hardware structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0034] Currently, table recognition methods in related technologies typically rely on computer vision algorithms or deep neural network models to extract lines from images, and then use these extracted lines to draw tables, thereby achieving table recognition in the image. However, in practical applications, table recognition algorithms face various interferences, such as image rotation due to shooting issues, partial occlusion of the table, incomplete table capture, and table distortion caused by paper wrinkles. The existing method of extracting table lines to draw tables for recognition is susceptible to problems such as image quality, which may result in discontinuous or incorrectly detected lines, leading to incorrect table structure and low recognition accuracy.

[0035] Based on this, the technical solution of the present invention identifies the table area in the image and identifies the endpoints of each cell and the connection relationship between adjacent endpoints in the table area. It no longer relies on line recognition, avoids table recognition errors caused by discontinuous table lines or detection errors, and improves the accuracy of table recognition.

[0036] According to an embodiment of the present invention, an embodiment of a table recognition method is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0037] This embodiment provides a table recognition method that can be used on the aforementioned mobile terminals, such as mobile phones, computers, and tablets. Figure 1 This is a flowchart of a table recognition method according to an embodiment of the present invention, such as... Figure 1 As shown, the process includes the following steps:

[0038] S11, acquire the image to be recognized.

[0039] The image to be recognized is an image containing a table. Electronic devices can acquire the image by responding to a user's screenshot command, by responding to a user's camera function call, or by responding to a user's upload command. Of course, other methods are also possible, but specific limitations are not discussed here.

[0040] S12, input the image to be recognized into the preset table detection model, and identify the table image from the image to be recognized.

[0041] The preset table detection model is a keypoint detection model based on YOLO V5. Specifically, sample images containing table images are collected for training. These sample images are used as input, and the sample table images and their positional correspondences within the sample images are used as output. This process trains the YOLO V5 model to obtain the table detection model. The positional correspondences represent the positions of the sample table images within the sample images and the endpoints of their bounding boxes.

[0042] The trained table detection model is deployed to an electronic device. When the electronic device acquires the image to be recognized, it inputs the image to be recognized into the table detection model to perform table detection and outputs the table image in the image to be recognized and the positional correspondence of the table image in the image to be recognized.

[0043] S13, input the table image into the preset table recognition model to determine the endpoint coordinates of each cell in the table image and the connection relationship between each endpoint.

[0044] The preset table recognition model is a keypoint recognition model based on the UNet model. Specifically, sample table images containing cells are acquired. The sample table image is used as input, and the endpoint coordinates of each cell and the connection relationships between the endpoints are used as output to train the UNet model, thus obtaining the table recognition model. The endpoint coordinates of each cell include the coordinate information of the center point of each endpoint region in the sample table image; the connection relationships between the endpoints are used to characterize whether two endpoints are adjacent endpoints within the same cell.

[0045] The trained table detection model is deployed to an electronic device. When the electronic device acquires the image to be recognized, it inputs the image to the table detection model to perform table detection and outputs the endpoint coordinates of each cell in the table image and the connection relationship between each endpoint, such as horizontal connection relationship, vertical connection relationship, etc.

[0046] S14, Generate the target table based on the endpoint coordinates and connection relationships.

[0047] The row and column positions of each cell are determined based on the endpoint coordinates of each cell, and the cell connection lines (such as horizontal and vertical connection lines) are generated based on the connection relationship between each endpoint. Then, the target table is generated based on the geometric relationship between each connection line.

[0048] The table recognition method provided in this embodiment determines the position of each cell by identifying the endpoint coordinates of each cell in a table image and the connection relationships between adjacent endpoints, thereby identifying the target table in the image. This method does not rely on the extraction of table lines; it achieves table recognition through endpoint extraction and endpoint connection relationship extraction, ensuring the accuracy of table structure recognition and improving the accuracy of table information recognition.

[0049] This embodiment provides a table recognition method that can be used on the aforementioned mobile terminals, such as mobile phones, computers, and tablets. Figure 2 This is a flowchart of a table recognition method according to an embodiment of the present invention, such as... Figure 2 As shown, the process includes the following steps:

[0050] S21, Obtain the image to be recognized. For detailed explanation, please refer to the relevant descriptions in the above embodiments; they will not be repeated here.

[0051] S22, input the image to be recognized into the preset table detection model, and identify the table image from the image to be recognized.

[0052] Specifically, step S22 above may include:

[0053] S221, Identify the image to be identified based on a preset table detection model, and obtain the table region and the corresponding border endpoints of the table region.

[0054] The border endpoints are the four endpoints that constitute the outer frame of the table. As mentioned above, the electronic device is equipped with a preset table detection model. The preset table detection model can perform table detection on the image to be recognized acquired by the electronic device, output the table region existing in the image to be recognized, and identify the endpoints of the outer frame of the table region, outputting the border endpoints corresponding to the table region.

[0055] S222, determine whether the table area meets the table attribute conditions based on the border endpoints.

[0056] The table attribute conditions are pre-defined attribute conditions for the table outline. Specifically, the outline polygon corresponding to the table area should be rectangular. The electronic device can generate the outline polygon corresponding to the table area based on the detected border endpoints, and compares this outline polygon with the table attribute conditions to determine whether it meets the attribute conditions for the table outline. If it does not meet the table attribute conditions, step S223 is executed; otherwise, the polygon formed by the border endpoints is used as the outline polygon of the table area.

[0057] S223, when the table area does not meet the table attribute conditions, correct the table area based on the border endpoints to obtain the table image.

[0058] When a table area does not meet the table attribute conditions, it indicates that the outer polygon corresponding to the table area has been distorted or has a rotation angle. In this case, the electronic device can construct the bounding rectangle of the table area by recognizing the boundary endpoints to correct the table area and obtain a corrected table image.

[0059] Specifically, step S223 above may include:

[0060] (1) Determine the minimum bounding rectangle corresponding to the table area based on the border endpoints.

[0061] (2) Perform perspective transformation based on the width and height of the minimum bounding rectangle to obtain the perspective transformation matrix from the endpoints of the border to the perspective transformation coordinates.

[0062] (3) Based on the perspective transformation matrix and the minimum bounding rectangle, the table region in the image to be recognized is corrected to obtain the table image.

[0063] In this embodiment, firstly, the minimum bounding rectangle is calculated based on the coordinates of the four border endpoints of the table area (denoted as src_p), and the width (denoted as img_crop_width) and height (denoted as img_crop_height) of the minimum bounding rectangle are determined. Secondly, a perspective transformation is performed based on the width and height of the minimum bounding rectangle, and the coordinates of the border endpoints after the perspective transformation (denoted as dst_p) are calculated. Specifically, the border endpoint coordinates can be represented as: [[0,0],[img_crop_width,0],[img_crop_width,img_crop_height],[0,img_crop_height]]. Next, the OpenCV function cv2.getPerspectiveTransform is called to obtain the perspective transformation matrix from coordinates src_p to coordinates dst_p. Then, the OpenCV function cv2.warpPerspective is called, and combined with the image to be recognized, the perspective transformation matrix, and the width and height of the minimum bounding rectangle, the corrected table image is obtained.

[0064] The outer frame of the table area is corrected by using the endpoints of the border to obtain a corrected table image, avoiding border distortion caused by factors such as paper wrinkles, and achieving accurate recognition of the table outer frame.

[0065] S23, input the table image into the preset table recognition model to determine the endpoint coordinates of each cell in the table image and the connection relationship between each endpoint.

[0066] Specifically, step S23 above may include:

[0067] S231, extract the endpoint region feature map and endpoint connection relationship feature map of the table image based on the preset table recognition model.

[0068] Endpoint region feature map is used to characterize the confidence feature map of the endpoint position of each cell in the table image, and endpoint connection relationship feature map is used to characterize the connection relationship between each cell endpoint and its adjacent cell endpoints.

[0069] As mentioned above, electronic devices are equipped with a preset table recognition model. This model can detect the center pixel of the endpoint region of each cell in a table image to determine the confidence level of the endpoint position of each cell. The pixel with higher confidence level is taken as the endpoint position of the cell, thereby obtaining the feature map of the endpoint region of each cell.

[0070] The endpoint connectivity feature map includes a horizontal connectivity feature map and a vertical connectivity feature map. While identifying endpoint positions, it checks whether each endpoint position and its adjacent cell endpoints are adjacent endpoints of the same cell in the horizontal direction, or whether each endpoint position and its adjacent cell endpoints are adjacent endpoints of the same cell in the vertical direction.

[0071] S232, perform pooling operation on the endpoint region feature map to obtain the endpoint region pooled feature map.

[0072] The electronic device performs max pooling on the endpoint region feature map (denoted as corner_map) of the cell to obtain the endpoint region pooled feature map (denoted as corner_pool_map), which is the pooled endpoint region feature map.

[0073] S233, extract the endpoint coordinates of each cell from the endpoint region pooling feature map.

[0074] The electronic device extracts pixel values ​​for endpoint locations from the endpoint region pooling feature map and then extracts pixel values ​​for endpoint locations from the endpoint region feature map. Next, it compares the pixel values ​​in the corner_map and corner_pool_map, and determines the endpoint coordinates of each cell based on the comparison results.

[0075] Specifically, step S233 above may include:

[0076] (1) Obtain the first pixel value of each endpoint region in the endpoint region feature map and the second pixel value of each endpoint region in the endpoint region pooling feature map.

[0077] (2) By comparing the first pixel value and the second pixel value, multiple target pixels with the same first pixel value and the second pixel value and both greater than the preset pixel value are identified.

[0078] (3) Sort the coordinates of multiple target pixels to obtain the endpoint coordinates of each cell.

[0079] The first pixel value is the pixel value of the endpoint location region in the endpoint region feature map; the second pixel value is the pixel value of the endpoint location region in the endpoint region pooling feature map. The electronic device can determine the endpoint position of each cell based on the confidence level of the endpoint position, and then extract the first pixel value from the endpoint region feature map and the corresponding second pixel value from the endpoint region pooling feature map based on this confidence level.

[0080] The first pixel value corresponding to the endpoint region is compared with the second pixel value to identify multiple pixels whose first pixel value and second pixel value are equal. Then, the pixel value corresponding to each pixel is compared with a preset pixel value to identify multiple target pixels whose pixel value is greater than the preset pixel value, and the coordinates of the multiple target pixels are determined. The preset pixel value is a pre-set endpoint pixel value, such as 0.5. The preset pixel value is not limited here, and those skilled in the art can determine it according to actual needs.

[0081] Determine the coordinates of each target pixel as endpoint coordinates, and sort the coordinates in a certain order (for example, using the top left corner of the table frame as the origin, sort the coordinates from left to right and from top to bottom; or, using the top left corner of the table frame as the origin, sort the coordinates from top to bottom and from left to right), and output the endpoint coordinates of each cell sorted by row and the endpoint coordinates of each cell sorted by column.

[0082] By detecting pixel values, the target pixel points that serve as cell endpoints are determined. Then, the pixel coordinates are sorted to obtain the endpoint coordinates of each row and column cell. Finally, the cell is restored based on the endpoint coordinates, which improves the cell recognition accuracy.

[0083] As an optional implementation, step (3) above may specifically include:

[0084] (31) Arrange the coordinates of multiple target pixels in rows according to a preset row sorting method to obtain the row sorting result.

[0085] (32) Traverse the ordinates of each target pixel in the row sorting result and determine whether the difference between the ordinates of two adjacent target pixels is less than the first preset threshold.

[0086] (33) When the difference between the ordinates of two adjacent target pixels is less than the first preset threshold, it is determined that the two adjacent target pixels are in the same row.

[0087] (34) Determine the endpoint coordinates of the starting cell and the ending cell of each row to obtain the endpoint coordinates of each cell sorted by row.

[0088] In this embodiment, firstly, the cells are sorted in ascending order according to their y-coordinates (x-coordinates). Secondly, they are sorted in ascending order according to their x-coordinates (x-coordinates). Thirdly, the endpoint coordinates of each sorted cell are traversed, and the difference between the y-coordinate of each cell's endpoint and the y-coordinate of its next adjacent cell is calculated. Next, this difference is compared with a first preset threshold to determine if it is less than the first preset threshold. If the difference is less than the first preset threshold, it indicates that two adjacent target pixels are in the same row, meaning that two adjacent endpoints belong to the same row of cells. The first preset threshold is a pre-defined difference in the y-coordinates of cells in the same row. Then, based on the endpoint coordinates of each row of cells, the starting and ending cell endpoint coordinates of each row of cells in the table image can be determined. Finally, the coordinate information of each cell endpoint sorted by row is output.

[0089] By arranging the coordinates of multiple target pixels using a preset row sorting method, the endpoint coordinates of each cell in the same row can be determined, thereby improving the accuracy of cell row recognition.

[0090] As an optional implementation, step (3) above may specifically include:

[0091] (41) Arrange the coordinates of multiple target pixels in columns according to the preset column sorting method to obtain the column sorting result.

[0092] (42) Traverse the x-coordinates of each target pixel in the column sorting results and determine whether the difference between the x-coordinates of two adjacent target pixels is less than the second preset threshold.

[0093] (43) When the difference between the ordinates of two adjacent target pixels is less than the second preset threshold, it is determined that the two adjacent target pixels are in the same column.

[0094] (44) Determine the endpoint coordinates of the starting cell and the ending cell of each column to obtain the endpoint coordinates of each cell sorted by column.

[0095] In this embodiment, firstly, the cells are sorted in ascending order by their x-coordinates (x) at the endpoints; secondly, they are sorted in ascending order by their y-coordinates (y) at the endpoints; thirdly, the endpoint coordinates of each sorted cell are traversed, and the difference between the x-coordinate of each cell's endpoint and the x-coordinate of its next adjacent cell is calculated; fourthly, this difference is compared with a second preset threshold to determine if it is less than the second preset threshold. If the difference is less than the second preset threshold, it indicates that two adjacent target pixels are in the same column, meaning that two adjacent endpoints belong to the same column of cells. The second preset threshold is a pre-set difference in the x-coordinates of cells in the same column. Subsequently, the starting and ending cell endpoint coordinates of each column of cells in the table image can be determined based on the endpoint coordinates of each row of cells; finally, the coordinate information of each cell endpoint sorted by column is output.

[0096] By arranging the coordinates of multiple target pixels using a preset column sorting method, the endpoint coordinates of each cell in the same column can be determined, thereby improving the accuracy of cell column recognition.

[0097] S234, perform binarization on the connection relationship feature map to obtain a binarized connection relationship feature map.

[0098] The binarized connectivity feature map includes a binarized horizontal connectivity feature map and a vertical connectivity feature map. Specifically, the electronic device binarizes the horizontal connectivity feature map of the cell (denoted as `hori_affinity_map`) to obtain a binarized horizontal connectivity feature map of adjacent cell endpoints (denoted as `hori_affinity_bin_map`). Simultaneously, it binarizes the vertical connectivity feature map of the cell (denoted as `vert_affinity_map`) to obtain a binarized vertical connectivity feature map of adjacent cell endpoints (denoted as `vert_affinity_bin_map`).

[0099] S235 extracts the connection relationships between each endpoint from the binary connection relationship feature map.

[0100] For the binarized horizontal connectivity feature map (hori_affinity_bin_map) and vertical connectivity feature map (vert_affinity_bin_map), Gaussian heatmaps are used to describe whether there are horizontal or vertical connectivity relationships between each cell endpoint and its adjacent cell endpoints. Specifically, if there is a horizontal or vertical connectivity relationship between two endpoints, a Gaussian heatmap exists between the two endpoints, with the two endpoints as the starting and ending points; if there is no horizontal or vertical connectivity relationship between two endpoints, no Gaussian heatmap exists between the two endpoints.

[0101] Specifically, step S235 above may include:

[0102] (1) Obtain the coordinates of adjacent endpoints in the binary connection relationship feature map.

[0103] (2) Based on the coordinates of adjacent endpoints, determine the set of connecting pixels between adjacent endpoints.

[0104] (3) When the average value of the pixels corresponding to the set of connected pixels is greater than the third preset threshold, it is determined that there is a connection between adjacent endpoints.

[0105] The connection relationships include horizontal connection relationships and vertical connection relationships.

[0106] The third preset threshold represents a pre-set pixel threshold when there is a connection relationship. The third preset threshold includes a horizontal connection relationship threshold and a vertical connection relationship threshold.

[0107] Specifically, the process of determining whether there is a lateral connection between adjacent endpoints is as follows:

[0108] First, iterate through the endpoint coordinates of each cell in the binarized horizontal connectivity feature map `hori_affinity_bin_map`, sorted by row, and compare the relationships between adjacent endpoint coordinates, designating the endpoint with the smaller x-coordinate as `hp1` and the endpoint with the larger x-coordinate as `hp2`. Second, obtain the set of pixels `hp_list` representing the horizontal connection between endpoints `hp1` and `hp2` from the binarized horizontal connectivity feature map `hori_affinity_bin_map`. Third, calculate the average horizontal pixel value of the set of pixels `hp_list`, denoted as `hp_score`. Then, compare the average horizontal pixel value `hp_score` with the horizontal connectivity threshold `hp_thr`. If the average horizontal pixel value `hp_score` is greater than the horizontal connectivity threshold `hp_thr`, it indicates that there is a horizontal connection between endpoints `hp1` and `hp2`; if the average horizontal pixel value `hp_score` is less than the horizontal connectivity threshold `hp_thr`, it indicates that there is no horizontal connection between endpoints `hp1` and `hp2`.

[0109] Specifically, the process of determining whether there is a vertical connection between adjacent endpoints is as follows:

[0110] First, iterate through the endpoint coordinates of each cell in the binarized vertical connectivity feature map vert_affinity_bin_map, sorted by column, and compare the relationships between adjacent endpoint coordinates, designating the endpoint with the smaller y-coordinate as vp1 and the endpoint with the larger y-coordinate as vp2. Second, obtain the set of vertical connection pixels vp_list located between endpoints vp1 and vp2 on the binarized vertical connectivity feature map vert_affinity_bin_map. Third, calculate the vertical pixel average value of the set of vertical connection pixels vp_list, denoted as vp_score. Then, compare the vertical pixel average value vp_score with the vertical connectivity threshold vp_thr. If the vertical pixel average value vp_score is greater than the vertical connectivity threshold vp_thr, it indicates that there is a vertical connection between endpoints vp1 and vp2; if the vertical pixel average value vp_score is less than the vertical connectivity threshold vp_thr, it indicates that there is no vertical connection between endpoints vp1 and vp2.

[0111] By detecting the average pixel value of the pixel set of the connecting pixels between adjacent endpoints, it is possible to determine whether there is a connection between adjacent endpoints. Thus, the corresponding horizontal and vertical connection relationships can be determined based on the endpoint coordinates, realizing the connection between adjacent endpoints and thereby achieving accurate cell recognition.

[0112] S24. Generate the target table based on the endpoint coordinates and connection relationships. For detailed explanations, please refer to the relevant descriptions in the above embodiments; they will not be repeated here.

[0113] The table recognition method provided in this embodiment detects whether the endpoints of the table region's borders meet table attribute conditions. When a table region that does not meet these conditions is detected, it is corrected to obtain a table region that does meet the conditions. This overcomes the problem of inaccurate table structure recognition caused by table image rotation or distortion, maximizing the accuracy of table image recognition. By extracting the endpoint coordinates of each cell and the connection relationships between them, each cell is generated based on these connections. This eliminates reliance on table lines, avoiding inaccurate table recognition caused by distorted or discontinuous lines.

[0114] This embodiment provides a table recognition method that can be used on the aforementioned mobile terminals, such as mobile phones, computers, and tablets. Figure 3 This is a flowchart of a table recognition method according to an embodiment of the present invention, such as... Figure 3 As shown, the process includes the following steps:

[0115] S31, Obtain the image to be recognized. For detailed explanation, please refer to the relevant descriptions in the above embodiments; they will not be repeated here.

[0116] S32, the image to be recognized is input into a preset table detection model, and the table image is identified from the image to be recognized. For detailed explanation, please refer to the relevant descriptions in the above embodiments, which will not be repeated here.

[0117] S33, input the table image into a preset table recognition model to determine the endpoint coordinates of each cell in the table image and the connection relationships between the endpoints. For detailed explanation, please refer to the relevant descriptions in the above embodiments; they will not be repeated here.

[0118] S34. Generate the target table based on the endpoint coordinates and connection relationships. For detailed explanations, please refer to the relevant descriptions in the above embodiments; they will not be repeated here.

[0119] S35, determine the target position of each cell in the target table.

[0120] The target position is used to characterize the row and column positions of each cell. Specifically, the electronic device can determine the row and column coordinates of each cell in the target table based on the endpoint coordinates of each cell, and then determine the target position of the cell in the target table by combining the row and column coordinates.

[0121] S36 checks the text information in each cell to obtain the text information.

[0122] The electronic device is equipped with a text detection model and a text recognition model. The text detection model is used to identify the text region corresponding to each cell, and this text detection model is a deep learning model based on network models such as dbNet and EAST. The text recognition model is used to identify the text information in the text region, and this text recognition model is a deep learning model based on network models such as CRNN and SVTR.

[0123] The text detection model detects text boxes in each cell, extracts text regions from the cells, and then uses a text recognition model to recognize the detected text regions, thus obtaining the text information in each cell.

[0124] S37, Generate target table information based on target location and text information.

[0125] The target positions of each cell in the target table and the corresponding text information of each cell are merged, and the text information is filled into the corresponding cells to generate the target table information corresponding to the table image.

[0126] The table recognition method provided in this embodiment identifies the corresponding text information by the target position of each cell, and determines the target table information based on the target position and text information, thereby realizing the matching of cells and their text information.

[0127] This embodiment also provides a table recognition device for implementing the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0128] This embodiment provides a table recognition device, such as... Figure 4 As shown, it includes:

[0129] The acquisition module 41 is used to acquire the image to be recognized.

[0130] The table detection module 42 is used to input the image to be recognized into a preset table detection model and identify the table image from the image to be recognized.

[0131] The table recognition module 43 is used to input the table image into the preset table recognition model to determine the endpoint coordinates of each cell in the table image and the connection relationship between each endpoint.

[0132] The generation module 44 is used to generate the target table based on the endpoint coordinates and connection relationships.

[0133] Optionally, the above-mentioned table detection module 42 specifically includes:

[0134] The endpoint acquisition submodule is used to identify the image to be identified based on a preset table detection model, and to obtain the table region and the corresponding border endpoints of the table region.

[0135] The attribute judgment submodule is used to determine whether a table area meets the table attribute conditions based on the border endpoints.

[0136] The correction submodule is used to correct the table area based on the border endpoints to obtain the table image when the table area does not meet the table attribute conditions.

[0137] Specifically, the above-mentioned correction submodule is used to: determine the minimum bounding rectangle corresponding to the table region based on the border endpoints; perform perspective transformation based on the width and height of the minimum bounding rectangle to obtain the perspective transformation matrix from the border endpoints to the perspective transformation coordinates; and correct the table region in the image to be recognized based on the perspective transformation matrix and the minimum bounding rectangle to obtain the table image.

[0138] Optionally, the above-mentioned table recognition module 43 specifically includes:

[0139] The first extraction submodule is used to extract the endpoint region feature map and endpoint connection relationship feature map of the table image based on the preset table recognition model.

[0140] The pooling submodule is used to perform pooling operations on the endpoint region feature map to obtain the endpoint region pooled feature map.

[0141] The second extraction submodule is used to extract the endpoint coordinates of each cell from the endpoint region pooling feature map.

[0142] The binarization submodule is used to binarize the connection relationship feature map to obtain a binarized connection relationship feature map.

[0143] The third extraction submodule is used to extract the connection relationships between each endpoint from the binary connection relationship feature map.

[0144] Specifically, the second extraction submodule is used to: obtain the first pixel value of each endpoint region in the endpoint region feature map and the second pixel value of each endpoint region in the endpoint region pooling feature map; compare the first pixel value and the second pixel value from the first pixel to determine multiple target pixels whose first pixel value and second pixel value are the same and both are greater than a preset pixel value; sort the coordinates of the multiple target pixels to obtain the endpoint coordinates of each cell.

[0145] Optionally, the second extraction submodule is further configured to: arrange the coordinates of multiple target pixels in a row according to a preset row sorting method to obtain a row sorting result; traverse the ordinates of each target pixel in the row sorting result to determine whether the difference between the ordinates of two adjacent target pixels is less than a first preset threshold; when the difference between the ordinates of two adjacent target pixels is less than the first preset threshold, determine that the two adjacent target pixels are in the same row; determine the endpoint coordinates of the starting cell and the endpoint coordinates of the ending cell of each row to obtain the endpoint coordinates of each cell sorted by row.

[0146] Optionally, the second extraction submodule is further configured to: arrange the coordinates of multiple target pixels in columns according to a preset column sorting method to obtain a column sorting result; traverse the x-coordinates of each target pixel in the column sorting result to determine whether the difference between the x-coordinates of two adjacent target pixels is less than a second preset threshold; when the difference between the y-coordinates of two adjacent target pixels is less than the second preset threshold, determine that the two adjacent target pixels are in the same column; determine the endpoint coordinates of the starting cell and the endpoint coordinates of the ending cell of each column to obtain the endpoint coordinates of each cell sorted by column.

[0147] Specifically, the third extraction submodule is used to: obtain the coordinates of adjacent endpoints in the binarized connectivity feature map; determine the set of connecting pixels between adjacent endpoints based on the coordinates of adjacent endpoints; and determine that there is a connectivity relationship between adjacent endpoints when the average pixel value corresponding to the set of connecting pixels is greater than a third preset threshold. The connectivity relationship includes horizontal connectivity and vertical connectivity.

[0148] Optionally, the above-mentioned form recognition device may further include:

[0149] The location determination module is used to determine the target location of each cell in the target table.

[0150] The text detection module is used to detect the text information in each cell and obtain the text information.

[0151] The table information generation module is used to generate target table information based on the target location and text information.

[0152] In this embodiment, the table recognition device is presented in the form of a functional unit. Here, a unit refers to an ASIC circuit, a processor and memory that execute one or more software or fixed programs, and / or other devices that can provide the above functions.

[0153] The further functional descriptions of each module and submodule are the same as those in the corresponding embodiments described above, and will not be repeated here.

[0154] The table recognition device provided in this embodiment determines the position of each cell by identifying the endpoint coordinates of each cell in a table image and the connection relationship between adjacent endpoints, thereby identifying the target table in the image. Therefore, table recognition is achieved through endpoint extraction and endpoint connection relationship extraction without relying on table line extraction, ensuring the accuracy of table structure recognition and improving the accuracy of table information recognition.

[0155] This invention also provides an electronic device having the above-described features. Figure 4 The table recognition device shown.

[0156] Please see Figure 5 , Figure 5 This is a schematic diagram of the structure of an electronic device provided in an optional embodiment of the present invention, such as... Figure 5 As shown, the electronic device may include: at least one processor 501, such as a central processing unit (CPU), at least one communication interface 503, memory 504, and at least one communication bus 502. The communication bus 502 is used to enable communication between these components. The communication interface 503 may include a display screen or a keyboard; optionally, the communication interface 503 may also include a standard wired interface or a wireless interface. The memory 504 may be high-speed volatile random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Optionally, the memory 504 may also be at least one storage device located remotely from the aforementioned processor 501. The processor 501 may be combined with... Figure 4 The described apparatus has an application program stored in memory 504, and a processor 501 calls the program code stored in memory 504 to perform any of the above method steps.

[0157] The communication bus 502 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The communication bus 502 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0158] The memory 504 may include volatile memory, such as random-access memory (RAM); the memory may also include non-volatile memory, such as flash memory, hard disk drive (HDD) or solid-state drive (SSD); the memory 504 may also include a combination of the above types of memory.

[0159] The processor 501 can be a central processing unit (CPU), a network processor (NP), or a combination of a CPU and an NP.

[0160] The processor 501 may further include a hardware chip. This hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The PLD may be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0161] Optionally, the memory 504 is also used to store program instructions. The processor 501 can invoke the program instructions to implement the table recognition method shown in the above embodiments of this application.

[0162] This invention also provides a non-transitory computer storage medium storing computer-executable instructions that can execute the table recognition method in any of the above method embodiments. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk drive (HDD), or solid-state drive (SSD), etc.; the storage medium may also include combinations of the above types of memory.

[0163] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A table recognition method, characterized in that, include: Acquire the image to be recognized; The image to be identified is input into a preset table detection model to identify the table image from the image to be identified; The table image is input into a preset table recognition model to determine the endpoint coordinates of each cell in the table image and the connection relationships between the endpoints. This includes: extracting endpoint region feature maps and endpoint connection relationship feature maps of the table image based on the preset table recognition model; performing pooling operations on the endpoint region feature maps to obtain endpoint region pooled feature maps; extracting the endpoint coordinates of each cell from the endpoint region pooled feature maps; performing binarization processing on the connection relationship feature maps to obtain binarized connection relationship feature maps; and extracting the connection relationships between the endpoints from the binarized connection relationship feature maps. Generate a target table based on the endpoint coordinates and the connection relationship; The preset table recognition model, while recognizing endpoint positions, detects whether each endpoint position and its adjacent cell endpoints are adjacent endpoints of the same cell in the horizontal or vertical direction.

2. The method according to claim 1, characterized in that, The step of inputting the image to be identified into a preset table detection model and identifying the table image from the image to be identified includes: Based on the preset table detection model, the image to be identified is recognized, and the table region and the corresponding border endpoints of the table region are obtained. Determine whether the table area meets the table attribute conditions based on the border endpoints; When the table area does not meet the table attribute conditions, the table area is corrected based on the border endpoints to obtain the table image.

3. The method according to claim 2, characterized in that, The step of correcting the table region based on the border endpoints to obtain the table image includes: The minimum bounding rectangle corresponding to the table area is determined based on the border endpoints; Based on the width and height of the minimum bounding rectangle, a perspective transformation is performed to obtain the perspective transformation matrix from the endpoints of the border to the perspective transformation coordinates; The table region in the image to be identified is corrected based on the perspective transformation matrix and the minimum bounding rectangle to obtain the table image.

4. The method according to claim 1, characterized in that, Extracting the endpoint coordinates of each cell from the endpoint region pooling feature map includes: Obtain the first pixel value of each endpoint region in the endpoint region feature map, and the second pixel value of each endpoint region in the endpoint region pooling feature map; By comparing the first pixel value and the second pixel value, multiple target pixels are identified whose first pixel value is the same as the second pixel value and whose values ​​are both greater than a preset pixel value. The coordinates of the multiple target pixels are sorted to obtain the endpoint coordinates of each cell.

5. The method according to claim 4, characterized in that, The step of sorting the coordinates of the plurality of target pixels to obtain the endpoint coordinates of each cell includes: The coordinates of the multiple target pixels are arranged in rows according to a preset row sorting method to obtain the row sorting result; Traverse the ordinates of each target pixel in the row sorting result and determine whether the difference between the ordinates of two adjacent target pixels is less than a first preset threshold. When the difference between the ordinates of two adjacent target pixels is less than the first preset threshold, it is determined that the two adjacent target pixels are in the same row. Determine the endpoint coordinates of the starting cell and the ending cell of each row to obtain the endpoint coordinates of each cell sorted by row.

6. The method according to claim 4 or 5, characterized in that, The step of sorting the coordinates of the plurality of target pixels to obtain the endpoint coordinates of each cell includes: The coordinates of the multiple target pixels are arranged in columns according to a preset column sorting method to obtain the column sorting result; Traverse the x-coordinates of each target pixel in the column sorting result, and determine whether the difference between the x-coordinates of two adjacent target pixels is less than a second preset threshold. When the difference in the horizontal coordinates of two adjacent target pixels is less than the second preset threshold, it is determined that the two adjacent target pixels are in the same column; Determine the endpoint coordinates of the starting cell and the ending cell of each column to obtain the endpoint coordinates of each cell sorted by column.

7. The method according to claim 1, characterized in that, Extracting the connection relationships between the endpoints from the binarized connection relationship feature map includes: Obtain the coordinates of adjacent endpoints in the binarized connectivity feature map; Based on the coordinates of the adjacent endpoints, determine the set of connecting pixels between the adjacent endpoints; When the average pixel value corresponding to the set of connected pixels is greater than a third preset threshold, it is determined that there is a connection relationship between the adjacent endpoints. The connection relationships include horizontal connection relationships and vertical connection relationships.

8. The method according to claim 1, characterized in that, Also includes: Determine the target position of each cell in the target table; The text information in each cell is detected to obtain the text information; Based on the target location and the text information, target table information is generated.

9. A form recognition device, characterized in that, include: The acquisition module is used to acquire the image to be recognized; The table detection module is used to input the image to be identified into a preset table detection model and identify the table image from the image to be identified; The table recognition module is used to input the table image into a preset table recognition model to determine the endpoint coordinates of each cell in the table image and the connection relationship between each endpoint; A generation module is used to generate a target table based on the endpoint coordinates and the connection relationship; The table recognition module includes: a first extraction submodule, used to extract endpoint region feature maps and endpoint connection relationship feature maps of the table image based on the preset table recognition model; a pooling submodule, used to perform pooling operations on the endpoint region feature maps to obtain endpoint region pooled feature maps; a second extraction submodule, used to extract the endpoint coordinates of each cell from the endpoint region pooled feature maps; a binarization submodule, used to perform binarization processing on the connection relationship feature maps to obtain binarized connection relationship feature maps; and a third extraction submodule, used to extract the connection relationships between the endpoints from the binarized connection relationship feature maps. The preset table recognition model, while recognizing endpoint positions, detects whether each endpoint position and its adjacent cell endpoints are adjacent endpoints of the same cell in the horizontal or vertical direction.

10. An electronic device, characterized in that, include: A memory and a processor are communicatively connected, the memory stores computer instructions, and the processor executes the table recognition method according to any one of claims 1-8 by executing the computer instructions.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing a computer to perform the table recognition method according to any one of claims 1-8.

Images