Method, device and readable storage medium for table recognition
By combining cropping and feature map processing with table detection, recognition, and text recognition models, the problem of low accuracy in table recognition in existing technologies has been solved, achieving high stability and high accuracy in restoring table information in complex scenarios.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- GLODON CO LTD
- Filing Date
- 2024-08-30
- Publication Date
- 2026-07-24
AI Technical Summary
Existing computer vision algorithms or deep neural network models for recognizing tables in images are easily affected by factors such as image quality, extreme aspect ratio cells, and inconspicuous table lines, resulting in low recognition accuracy.
By acquiring the image to be recognized, the table region is cropped using the trained table detection model. The feature map of the pixel coordinates, endpoint coordinates and offset distance of the cell center region is obtained by combining the table recognition model to determine the cell position information. Finally, the table text information is extracted by combining the text recognition model to generate the table information.
It improves the stability and accuracy of table recognition in interference scenarios and can accurately restore the original structure of the table.
Smart Images
Figure CN119007232B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image processing technology, and more specifically to a method, apparatus, device, and readable storage medium for table recognition. Background Technology
[0002] With the development of computer technology, text recognition has been widely applied in various fields, and the structured recognition of tables in images is one of the key tasks in text recognition. Existing table recognition methods typically extract lines from images based on traditional computer vision algorithms or deep neural network models, and then use the extracted lines to draw tables to achieve table recognition in images.
[0003] However, in real-world applications, the quality of the table images to be recognized is inconsistent. Issues may include partially obscured tables, incomplete images, paper wrinkles causing distortion, cells with extreme aspect ratios, and indistinct table lines. Current methods for table recognition, which extract table lines to create the table, are susceptible to problems due to image quality, extreme aspect ratios, and indistinct lines. This can lead to discontinuous lines or detection errors, resulting in incorrect table structure and low recognition accuracy.
[0004] There is currently no effective solution to the technical problem of low accuracy in recognizing tables in images based on traditional computer vision algorithms or deep neural network models. Summary of the Invention
[0005] The purpose of this invention is to provide a method, apparatus, device, and readable storage medium for table recognition, which can solve the technical problem that the accuracy of table recognition in images based on traditional computer vision algorithms or deep neural network models is low.
[0006] One aspect of the present invention provides a method for table recognition, the method comprising: acquiring an image to be recognized, wherein the image to be recognized includes table information and non-table information; inputting the image to be recognized into a trained table detection model to obtain position information of table regions in the image to be recognized; cropping the image to be recognized using the position information of the table regions to obtain a table image to be recognized; inputting the table image to be recognized into a trained table recognition model to obtain pixel coordinates of cell center regions, cell endpoint coordinates, and offset distance feature maps of the image to be recognized, wherein the offset distance feature maps characterize the offset relationship between pixel coordinates of cell center regions and cell endpoint coordinates; determining the position information of cells in the table image to be recognized using the pixel coordinates of cell center regions, cell endpoint coordinates, and offset distance feature maps of the image to be recognized; inputting the table image to be recognized and the position information of the cells into a trained text recognition model to obtain text information of the cells in the table image to be recognized; and generating table information corresponding to the table image to be recognized using the position information of the cells and the text information of the cells in the table image to be recognized.
[0007] Optionally, the table image to be recognized is input into the trained table recognition model to obtain the pixel coordinates of the cell center region, the cell endpoint coordinates, and the offset distance feature map of the image to be recognized. This includes: extracting feature information of the table image to be recognized through a downsampling module, and decoding the feature information of the table image to be recognized through an upsampling module to obtain the cell center region feature map, endpoint region feature map, and offset distance feature map of the table image to be recognized; performing max pooling on the center region feature map and the endpoint region feature map respectively to obtain the pooled center region feature map and the pooled endpoint region feature map; obtaining the same pixel values between the pooled center region feature map and the center region feature map, and the same pixel values between the pooled endpoint region feature map and the endpoint region feature map; using the pixels with the same pixel values greater than a first preset threshold as the cell center region pixel coordinates and cell endpoint coordinates respectively, and sorting the cell endpoints according to a preset order.
[0008] Optionally, the position information of cells in the table image to be recognized is determined by using the pixel coordinates of the cell center region, the cell endpoint coordinates, and the offset distance feature map of the cell in the image to be recognized. This includes: determining the cell coordinates corresponding to each central region pixel using the pixel coordinates of the cell center region and the offset distance feature map; calculating the distance between the cell coordinates corresponding to the central region pixel and the cell endpoint coordinates; filtering cell coordinates whose distance is less than or equal to a second preset threshold; updating the filtered cell coordinates using the cell endpoint coordinates whose distance is less than or equal to the second preset threshold; determining whether the updated cell coordinates corresponding to the central region pixel match the cell endpoint coordinates; if yes, using the updated cell coordinates corresponding to the central region pixel as the position information of the cell in the table image to be recognized; if no, extracting the coordinates of points without matching results from the updated cell coordinates corresponding to the central region pixel, calculating the spatial distance between the point coordinates and the cell endpoint coordinates, updating the point coordinates again using the endpoint with the minimum spatial distance to the point coordinates, and using the updated cell coordinates corresponding to the central region pixel as the position information of the cell in the table image to be recognized.
[0009] Optionally, after determining the position information of the cell in the table image to be recognized by using the pixel coordinates of the cell center region, the cell endpoint coordinates, and the offset distance feature map of the image to be recognized, the method further includes: determining the border line cluster of the table based on the position information of the cell, wherein the border line cluster of the table includes horizontal line clusters and vertical line clusters; and determining the row and column indices of the cell in the table image to be recognized by using the border line clusters of the table and the position information of the cell, wherein the row and column indices include the starting row index, the starting column index, the ending row index, and the ending column index.
[0010] Optionally, determining the table's border line clusters based on cell location information includes: calculating a first spatial distance between the top-right corner coordinates of a cell and the top-left corner coordinates of other cells, and a second spatial distance between the bottom-right corner coordinates of a cell and the bottom-left corner coordinates of other cells; generating left-right adjacent cell clusters for cells whose first and second spatial distances are both less than a third preset threshold; calculating a third spatial distance between the bottom-left corner coordinates of a cell and the top-left corner coordinates of other cells, and a fourth spatial distance between the bottom-right corner coordinates of a cell and the top-right corner coordinates of other cells; generating top-bottom adjacent cell clusters for cells whose third and fourth spatial distances are both less than the third preset threshold; connecting the first-direction border lines of the left-right adjacent cell clusters according to the order of the cells in the first direction, merging the connecting lines whose distance is less than the third preset threshold, and sorting the merged border lines according to the second direction to generate the table's horizontal line clusters; connecting the second-direction border lines of the top-bottom adjacent cell clusters according to the order of the cells in the second direction, merging the connecting lines whose distance is less than the third preset threshold, and sorting the merged border lines according to the first direction to generate the table's vertical line clusters.
[0011] Optionally, the row and column indices of cells in the table image to be recognized are determined by the position information of the table's border line clusters and cells. This includes: determining the feature points of the cell's first-direction border line, calculating the distance between the feature points of the first-direction border line and each horizontal line of the table's horizontal line cluster; selecting the index of the horizontal line with the smallest distance as the cell's starting row index or ending row index; determining the feature points of the cell's second-direction border line, calculating the distance between the feature points of the second-direction border line and each vertical line of the table's vertical line cluster; and selecting the index of the vertical line with the smallest distance as the cell's starting column index or ending column index.
[0012] Another aspect of the present invention provides a table recognition apparatus, comprising: an acquisition module for acquiring an image to be recognized, wherein the image to be recognized includes table information and non-table information; a cropping module for inputting the image to be recognized into a trained table detection model to obtain position information of table regions in the image to be recognized, and cropping the image to be recognized using the position information of the table regions to obtain a table image to be recognized; a table recognition module for inputting the table image to be recognized into a trained table recognition model to obtain pixel coordinates of the center region of cells, coordinates of cell endpoints, and an offset distance feature map of the image to be recognized, wherein the offset distance feature map characterizes the offset relationship between the pixel coordinates of the center region of cells and the coordinates of cell endpoints; a determination module for determining the position information of cells in the table image to be recognized using the pixel coordinates of the center region of cells, coordinates of cell endpoints, and the offset distance feature map of the image to be recognized; and a text recognition module for inputting the table image to be recognized and the position information of cells into a trained text recognition model to obtain text information of cells in the table image to be recognized, and generating table information corresponding to the table image to be recognized using the position information of cells and the text information of cells in the table image to be recognized.
[0013] Optionally, the table recognition module is specifically used to: extract feature information of the table image to be recognized through the downsampling module, and decode the feature information of the table image to be recognized through the upsampling module to obtain the center region feature map, endpoint region feature map, and offset distance feature map of the cell of the table image to be recognized; perform max pooling operation on the center region feature map and the endpoint region feature map respectively to obtain the pooled center region feature map and the pooled endpoint region feature map; obtain the same pixel values between the pooled center region feature map and the center region feature map, and the same pixel values between the pooled endpoint region feature map and the endpoint region feature map; take the pixels with the same pixel values greater than a first preset threshold as the coordinates of the cell center region pixel point and the cell endpoint coordinates respectively, and sort the cell endpoints according to a preset order.
[0014] Another aspect of the present invention provides a computer device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the table recognition method of any of the above embodiments.
[0015] Another aspect of the present invention provides a computer storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the table recognition method of any of the above embodiments. Further, the computer-readable storage medium may primarily include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application program required for at least one function, etc.; and the data storage area may store data created based on the use of blockchain nodes, etc.
[0016] This invention obtains the endpoint coordinates of each cell in a table, the coordinates of the central pixel of each cell, and the coordinate offset between each pixel in the central region and the corresponding cell endpoint, thereby determining the position information of each cell. Combined with the recognized table text information, the original structure information of the table is restored, thus improving the stability and accuracy of table recognition in interference scenarios. Attached Figure Description
[0017] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:
[0018] Figure 1 An optional flowchart of the table recognition method provided in Embodiment 1 of the present invention is shown;
[0019] Figure 2 A structural block diagram of the table recognition device provided in Embodiment 2 of the present invention is shown; and
[0020] Figure 3 A block diagram of a computer device suitable for implementing a table recognition method according to Embodiment 3 of the present invention is shown. Detailed Implementation
[0021] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without inventive effort are within the scope of protection of this invention.
[0022] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.
[0023] Example 1
[0024] This embodiment provides a method for table recognition. Figure 1 A flowchart of the method for identifying the table is shown, such as Figure 1 As shown, the method for table recognition may include steps S101 to S105, wherein:
[0025] Step S101: Obtain the image to be recognized, wherein the image to be recognized contains tabular information and non-tabular information;
[0026] In practical applications, the image to be identified can be an image containing at least table information. It can be obtained from local storage space or from a webpage through web crawling technology. No restrictions are imposed here.
[0027] Step S102: Input the image to be recognized into the trained table detection model to obtain the position information of the table region in the image to be recognized. Crop the image to be recognized using the position information of the table region to obtain the table image to be recognized.
[0028] The table detection model can be the YOLO V8 detection model, which is trained based on sample table images and the positional relationship between sample table images and sample images.
[0029] The location information of the table area can be the coordinates of the four endpoints of the outer rectangle of the table or the coordinates of any two opposite corner endpoints.
[0030] Based on the location information of the table area obtained above, the image to be recognized is cropped to obtain the cropped table image to be recognized.
[0031] Step S103: Input the table image to be recognized into the trained table recognition model to obtain the pixel coordinates of the cell center region, the cell endpoint coordinates, and the offset distance feature map of the image to be recognized. The offset distance feature map represents the offset relationship between the pixel coordinates of the cell center region and the cell endpoint coordinates.
[0032] The table recognition model can be a UNet model. The table recognition model is trained based on sample table images and the coordinates of the center pixels of each cell, the coordinates of the endpoints of each cell, and the offset relationship between the center pixels and the endpoints of each cell. Specifically, the center region of each cell in the sample table image includes the pixel coordinates contained in the central Gaussian plot region generated based on the cell center point coordinates and the cell aspect ratio. The endpoint coordinates of each cell in the sample table image are the coordinates of the endpoints of each cell within the sample table image. The offset distance between the center region and the endpoints of each cell includes the coordinate differences between the pixels in the center region and the four endpoints of the corresponding cell.
[0033] The trained table recognition model can accurately identify table feature information by recognizing the table image.
[0034] Step S104: Determine the position information of the cells in the table image to be identified by using the pixel coordinates of the center region of the cells in the image to be identified, the coordinates of the cell endpoints, and the offset distance feature map.
[0035] The feature map of the pixel coordinates of the cell center area, the coordinates of the endpoints, and the offset distance between the pixel coordinates of the cell center area and the coordinates of the cell endpoints represents the position of the table cell from multiple dimensions, which helps to restore the accurate table structure and facilitate the effective extraction of the table text content.
[0036] Step S105: Input the position information of the table image to be recognized and the cells into the trained text recognition model to obtain the text information of the cells in the table image to be recognized. Generate the table information corresponding to the table image to be recognized by using the position information of the cells and the text information of the cells in the table image to be recognized.
[0037] Inputting the image of the table to be recognized and the position information of the cells into the trained text recognition model to recognize the text content of the table is beneficial to the accurate positioning of the table text during the recognition process. Based on the recognized table text and the calculated cell position information in the table image, the original table information can be restored, which improves the stability and accuracy of table recognition in interference scenarios.
[0038] Preferably, step S103 may include steps S1031 to S1034, wherein:
[0039] Step S1031: Extract feature information of the table image to be identified through the downsampling module, and decode the feature information of the table image to be identified through the upsampling module to obtain the feature map of the center region, the feature map of the endpoint region, and the feature map of the offset distance of the cell of the table image to be identified.
[0040] The central region feature map has 1 channel and is represented by a Gaussian heatmap, which describes the central region of each cell, i.e., the confidence level of the cell center point at each location.
[0041] The endpoint region feature map has 1 channel, which describes the confidence level of each position in the table image as a cell endpoint.
[0042] The feature map with 8 channels describes the difference between the x-coordinate of the pixel in the center region of each cell and the x-coordinate of each cell endpoint, as well as the difference between the y-coordinate of the pixel in the center region of each cell and the y-coordinate of each cell endpoint. Specifically, the 8 channels correspond to the differences in x-coordinate between the pixel in the center region and the corresponding top-left corner (endpoint) of the cell, the differences in y-coordinate between the pixel in the center region and the corresponding top-left corner (endpoint) of the cell, the differences in x-coordinate between the pixel in the center region and the corresponding top-right corner (endpoint) of the cell, the differences in y-coordinate between the pixel in the center region and the corresponding top-right corner (endpoint) of the cell, the differences in x-coordinate between the pixel in the center region and the corresponding bottom-right corner (endpoint) of the cell, the differences in x-coordinate between the pixel in the center region and the corresponding bottom-right corner (endpoint) of the cell, and the differences in y-coordinate between the pixel in the center region and the corresponding bottom-left corner (endpoint) of the cell.
[0043] Step S1032: Perform max pooling on the central region feature map and the endpoint region feature map respectively to obtain the pooled central region feature map and the pooled endpoint region feature map.
[0044] Step S1033: Obtain the same pixel values between the pooled center region feature map and the center region feature map, and the same pixel values between the pooled endpoint region feature map and the endpoint region feature map.
[0045] Step S1033: Pixels with the same pixel value greater than the first preset threshold are respectively used as the pixel coordinates of the center area of the cell and the cell endpoint coordinates, and the cell endpoints are sorted according to a preset order.
[0046] The first preset threshold can be 0.5. The preset order can be the position order of the endpoint coordinates of each cell in the table image to be recognized, such as sorting them from left to right and from top to bottom.
[0047] Specifically, the table recognition model is based on a UNet model structure. First, image features are extracted using a downsampling module, followed by decoding using an upsampling module. This yields the coordinates of the endpoints of each cell in the cropped table image, the pixel coordinates of the center region of each cell, and the offset relationship between the pixel coordinates of the center region of each cell and the endpoints. The endpoint coordinates of each cell in the cropped table image can be understood as the coordinates of the center pixel of the endpoint region of each cell within the cropped table image. The pixel coordinates of the center region of each cell in the cropped table image can be understood as mapping a 2D Gaussian heatmap onto each cell, where the center point coordinates of the cell are set to x0, y0, the cell width is w, and the cell height is h. The distribution of the Gaussian kernel is then:
[0048]
[0049] Where σ x =αw / 6, σ y =α h / 6, α can be set to 0.54.
[0050] The offset relationship between the center pixel coordinates of each cell and the endpoints of each cell can be the difference between the x-coordinate of the center pixel and the x-coordinate of the endpoints of each cell, and the difference between the y-coordinate of the center pixel and the y-coordinate of the endpoints of each cell.
[0051] Preferably, step S104 may include steps S1041 to S1045, wherein:
[0052] Step S1041: Determine the cell coordinates corresponding to each central region pixel using the cell center region pixel coordinates and offset distance feature map;
[0053] After obtaining the cell coordinates corresponding to the pixels in each central region, cells with an IoU greater than the preset IoU threshold (which can be 0.5) can be merged by taking the average of the coordinates of the aforementioned cells.
[0054] Step S1042: Calculate the distance between the cell coordinates corresponding to the pixel in the central region and the cell endpoint coordinates, filter the cell coordinates whose distance is less than or equal to the second preset threshold, and update the filtered cell coordinates using the cell endpoint coordinates whose distance is less than or equal to the second preset threshold.
[0055] The second preset threshold can be 2.
[0056] Step S1043: Determine whether the cell coordinates corresponding to the updated center region pixels match the cell endpoint coordinates;
[0057] Step S1044: If yes, use the cell coordinates corresponding to the updated central region pixels as the cell position information in the table image to be identified.
[0058] Step S1045: If not, extract the coordinates of points without matching results from the cell coordinates corresponding to the updated center pixel, calculate the spatial distance between the point coordinates and the cell endpoint coordinates, update the point coordinates again using the endpoint with the minimum spatial distance from the point coordinates, and use the cell coordinates corresponding to the updated center pixel as the position information of the cell in the table image to be identified.
[0059] Iterate through the cell coordinates corresponding to the updated center pixel, and determine whether the endpoint coordinates in each cell match the coordinates of the endpoints of each cell. If an endpoint in a cell does not match the coordinates of the endpoints of each cell, calculate whether there is a spatial distance between the endpoint and the coordinates of the endpoints of each cell that is less than a set threshold (e.g., 30). If the difference between their x-coordinates is less than the set threshold (set as the width of all cells × ratio, where ratio is set to 0.1) and the difference between their y-coordinates is less than the set threshold (set as the height of all cells × ratio, where ratio is set to 0.3), and a match is found with the cell endpoints, then take the cell endpoint with the closest spatial distance and update it again. The finally updated phrase is output as the cell position information of the table image to be recognized, according to each coordinate.
[0060] Preferably, the method further includes steps A1 to A2, wherein:
[0061] Step A1: Determine the table border line clusters based on the cell position information. The table border line clusters include horizontal line clusters and vertical line clusters.
[0062] Step A2: Determine the row and column indices of the cells in the table image to be recognized by using the border line clusters and cell position information of the table. The row and column indices include the starting row index, the starting column index, the ending row index, and the ending column index.
[0063] By determining the row and column indices of a cell within the image of the table to be recognized, the cell's exact row and column can be pinpointed, which helps improve the accuracy of table information recognition.
[0064] Preferably, step A1 may include steps A11 to A16, wherein:
[0065] Step A11: Calculate the first spatial distance between the top right corner coordinate of the cell and the top left corner coordinate of other cells, and the second spatial distance between the bottom right corner coordinate of the cell and the bottom left corner coordinate of other cells;
[0066] Step A12: Generate a cluster of cells with a first spatial distance and a second spatial distance that are both less than a third preset threshold.
[0067] The third preset threshold can be 10.
[0068] First, retrieve all left and right adjacent cells. This is done by determining if the spatial distance between the top-right corner of a cell and the top-left corner of another cell is less than a set threshold (e.g., 10), and simultaneously, the spatial distance between the bottom-right corner of a cell and the bottom-left corner of another cell is less than a set threshold (e.g., 10). If these conditions are met, the cells are considered left and right adjacent. Then, retrieve all clusters of left and right adjacent cells, where each element in a cluster is a cell connected in the x-direction.
[0069] Step A13: Calculate the third spatial distance between the bottom left corner coordinate of the cell and the top left corner coordinate of other cells, and the fourth spatial distance between the bottom right corner coordinate of the cell and the top right corner coordinate of other cells;
[0070] Step A14: Generate clusters of cells with both the third spatial distance and the fourth spatial distance less than the third preset threshold.
[0071] Next, retrieve all vertically adjacent cells. The criteria for this are: if the spatial distance between the bottom-left corner of a cell and the top-left corner of another cell is less than a set threshold (e.g., 10), and simultaneously, the spatial distance between the bottom-right corner of a cell and the top-right corner of another cell is less than a set threshold (e.g., 10), then the pair of cells are considered vertically adjacent. Then, retrieve all vertically adjacent cell clusters, where each element in a cluster is a cell connected in the y-direction.
[0072] Step A15: Connect the first-direction border lines of adjacent cell clusters in the order of the cells in the first direction, merge the connecting lines with a distance less than the third preset threshold, and sort the merged border lines in the second direction to generate the horizontal line clusters of the table.
[0073] The first direction of the border line can be the top and bottom edges of the cell. The first direction can be the x-direction. The second direction can be the y-direction.
[0074] Iterate through each element in the clusters of adjacent cells on the left and right, and obtain the top and bottom edges of the cells in these elements. Connect the top and bottom edges that are connected end to end, merge lines whose distance between each pair of line segments is less than a set threshold (e.g., 10), and sort them according to the y-coordinate of the center point of the line segments in ascending order, outputting the horizontal line clusters of the table.
[0075] Step A16: Connect the second-direction border lines of adjacent cell clusters in the order of the cells in the second direction, merge the connecting lines with a distance less than a third preset threshold, and sort the merged border lines in the first direction to generate the vertical line clusters of the table.
[0076] Iterate through each element in the clusters of adjacent cells above and below, and obtain the left and right edge lines of the cells in these elements. By merging the left / right edge lines that are connected end to end, and merging the lines whose distance between each pair of line segments is less than a set threshold (e.g., 10), and sorting them according to the x-coordinate of the center point of the line segments in ascending order, the output is a cluster of vertical lines in the table.
[0077] Preferably, step A2 may include steps A21 to A24, wherein:
[0078] Step A21: Determine the feature points of the first direction border line of the cell, and calculate the distance between the feature points of the first direction border line and each horizontal line of the table's horizontal line cluster.
[0079] The feature points of the first direction border line can be the left endpoint, center point, and right endpoint of the top edge of the cell, and the left endpoint, center point, and right endpoint of the bottom edge.
[0080] Step A22: Select the index of the horizontal line with the smallest distance as the starting or ending row index of the cell;
[0081] Calculate the distances from the left endpoint, center point, and right endpoint of the top edge of each cell to each horizontal line in the table's horizontal line cluster, and select the index of the horizontal line with the smallest distance as the starting row index of that cell;
[0082] Calculate the distances from the left endpoint, center point, and right endpoint of the bottom edge of each cell to each horizontal line in the table's horizontal line cluster. Select the index of the horizontal line with the smallest distance and record it as the ending row index of that cell.
[0083] Step A23: Determine the feature points of the second-direction border line of the cell, and calculate the distance between the feature points of the second-direction border line and each vertical line of the table's vertical line cluster.
[0084] The feature points of the second-direction border line can be the upper endpoint, center point, and lower endpoint of the left edge line of the cell, and the upper endpoint, center point, and lower endpoint of the right edge line.
[0085] Step A24: Select the index of the vertical line with the smallest distance as the starting column index or ending column index of the cell.
[0086] Calculate the distances from the top, center, and bottom points of the left edge of each cell to each vertical line in the table's vertical line cluster, and select the index of the vertical line with the smallest distance as the starting column index of that cell;
[0087] Calculate the distances from the top, center, and bottom points of the right edge of each cell to each vertical line in the table's horizontal line clusters. Select the index of the vertical line with the smallest distance and denote it as the terminal column index of that cell.
[0088] This embodiment obtains the endpoint coordinates of each cell in the table, the pixel coordinates of the center area of each cell, and the coordinate offset between each pixel in the center area and the corresponding cell endpoint, thereby determining the position information of each cell. Combined with the recognized table text information, the original structure information of the table is restored, thus improving the stability and accuracy of table recognition in interference scenarios.
[0089] Example 2
[0090] Embodiment 2 of the present invention also provides a table recognition device, which corresponds to the table recognition method provided in Embodiment 1 above. The corresponding technical features and effects will not be detailed in this embodiment; relevant aspects can be referred to in Embodiment 1 above. Specifically, Figure 2 A structural block diagram of the device identified by the table is shown. For example... Figure 2 As shown, the table recognition device 200 includes an acquisition module 201, a cropping module 202, a table recognition module 203, a determination module 204, and a text recognition module 205, wherein:
[0091] The acquisition module 201 is used to acquire the image to be recognized, wherein the image to be recognized includes tabular information and non-tabular information;
[0092] The cropping module 202, connected to the acquisition module 201, is used to input the image to be recognized into the trained table detection model, obtain the position information of the table region in the image to be recognized, and crop the image to be recognized using the position information of the table region to obtain the table image to be recognized.
[0093] The table recognition module 203, connected to the cropping module 202, is used to input the table image to be recognized into the trained table recognition model to obtain the pixel coordinates of the cell center region, the cell endpoint coordinates, and the offset distance feature map of the image to be recognized. The offset distance feature map represents the offset relationship between the pixel coordinates of the cell center region and the cell endpoint coordinates.
[0094] The determination module 204, connected to the table recognition module 203, is used to determine the position information of the cells in the table image to be recognized by using the pixel coordinates of the center region of the cells, the coordinates of the cell endpoints, and the offset distance feature map of the image to be recognized.
[0095] The text recognition module 205, connected to the determination module 204, is used to input the position information of the table image to be recognized and the cells into the trained text recognition model to obtain the text information of the cells in the table image to be recognized, and to generate the table information corresponding to the table image to be recognized by using the position information of the cells and the text information of the cells in the table image to be recognized.
[0096] Optionally, the table recognition module is specifically used to: extract feature information of the table image to be recognized through the downsampling module, and decode the feature information of the table image to be recognized through the upsampling module to obtain the center region feature map, endpoint region feature map, and offset distance feature map of the cell of the table image to be recognized; perform max pooling operation on the center region feature map and the endpoint region feature map respectively to obtain the pooled center region feature map and the pooled endpoint region feature map; obtain the same pixel values between the pooled center region feature map and the center region feature map, and the same pixel values between the pooled endpoint region feature map and the endpoint region feature map; take the pixels with the same pixel values greater than a first preset threshold as the coordinates of the cell center region pixel point and the cell endpoint coordinates respectively, and sort the cell endpoints according to a preset order.
[0097] Optionally, the determining module is specifically used for: determining the cell coordinates corresponding to each central region pixel using the cell center region pixel coordinates and offset distance feature map; calculating the distance between the cell coordinates corresponding to the central region pixel and the cell endpoint coordinates, filtering cell coordinates whose distance is less than or equal to a second preset threshold, and updating the filtered cell coordinates using the cell endpoint coordinates whose distance is less than or equal to the second preset threshold; determining whether the updated cell coordinates corresponding to the central region pixel match the cell endpoint coordinates; if yes, using the updated cell coordinates corresponding to the central region pixel as the cell position information in the table image to be recognized; if no, extracting the point coordinates without matching results from the updated cell coordinates corresponding to the central region pixel, calculating the spatial distance between the point coordinates and the cell endpoint coordinates, updating the point coordinates again using the endpoint with the minimum spatial distance to the point coordinates, and using the updated cell coordinates corresponding to the central region pixel as the cell position information in the table image to be recognized.
[0098] Optionally, the device further includes a row and column index determination module, comprising: a bounding box cluster determination unit, used to determine the border line clusters of the table based on the cell position information, wherein the border line clusters of the table include horizontal line clusters and vertical line clusters; and a row and column index determination unit, used to determine the row and column index of the cell in the table image to be recognized by means of the table border line clusters and the cell position information, wherein the row and column index includes a starting row index, a starting column index, an ending row index, and an ending column index.
[0099] Optionally, the bounding box cluster determination unit is specifically used for: calculating the first spatial distance between the upper right corner coordinate of a cell and the upper left corner coordinate of other cells, and the second spatial distance between the lower right corner coordinate of a cell and the lower left corner coordinate of other cells; generating left-right adjacent cell clusters for cells whose first and second spatial distances are both less than a third preset threshold; calculating the third spatial distance between the lower left corner coordinate of a cell and the upper left corner coordinate of other cells, and the fourth spatial distance between the lower right corner coordinate of a cell and the upper right corner coordinate of other cells; generating vertically adjacent cell clusters for cells whose third and fourth spatial distances are both less than the third preset threshold; connecting the first direction border lines of the left-right adjacent cell clusters according to the order of the cells in the first direction, merging the connecting lines whose distance is less than the third preset threshold, and sorting the merged border lines according to the second direction to generate the horizontal line clusters of the table; connecting the second direction border lines of the vertically adjacent cell clusters according to the order of the cells in the second direction, merging the connecting lines whose distance is less than the third preset threshold, and sorting the merged border lines according to the first direction to generate the vertical line clusters of the table.
[0100] Optionally, the row and column index determination unit is specifically used for: determining the feature points of the first direction border line of the cell, calculating the distance between the feature points of the first direction border line and each horizontal line of the table's horizontal line cluster; selecting the index of the horizontal line with the smallest distance as the starting row index or ending row index of the cell; determining the feature points of the second direction border line of the cell, calculating the distance between the feature points of the second direction border line and each vertical line of the table's vertical line cluster; and selecting the index of the vertical line with the smallest distance as the starting column index or ending column index of the cell.
[0101] Example 3
[0102] Figure 3 A block diagram of a computer device suitable for implementing a table recognition method according to Embodiment 3 of the present invention is shown. In this embodiment, the computer device 300 may be a smartphone, tablet computer, laptop computer, desktop computer, rack server, blade server, tower server, or cabinet server (including independent servers or server clusters composed of multiple servers), etc., executing a program. Figure 3As shown, the computer device 300 in this embodiment includes, but is not limited to, a memory 301, a processor 302, and a network interface 303 that are communicatively connected to each other via a system bus. It should be noted that... Figure 3 Only a computer device 300 with components 301-303 is shown; however, it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.
[0103] In this embodiment, the memory 303 includes at least one type of computer-readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 301 may be an internal storage unit of the computer device 300, such as the hard disk or memory of the computer device 300. In other embodiments, the memory 301 may also be an external storage device of the computer device 300, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device 300. Of course, the memory 301 may also include both the internal storage unit and the external storage device of the computer device 300. In this embodiment, the memory 301 is typically used to store the operating system and various application software installed on the computer device 300, such as the program code of a table recognition method.
[0104] In some embodiments, processor 302 may be a central processing unit (CPU), controller, microcontroller, microprocessor, or other data processing chip. This processor 302 is typically used to control the overall operation of computer device 300. For example, it performs control and processing related to data interaction or communication with computer device 300. In this embodiment, processor 302 is used to run program code for the steps of a table recognition method stored in memory 301.
[0105] In this embodiment, the table recognition method stored in memory 301 can also be divided into one or more program modules and executed by one or more processors (processor 302 in this embodiment) to complete the present invention.
[0106] Network interface 303 may include a wireless network interface or a wired network interface, which is typically used to establish a communication link between computer device 300 and other computer devices. For example, network interface 303 is used to connect computer device 300 to an external terminal via a network, establishing a data transmission channel and communication link between computer device 300 and the external terminal. The network may be an intranet, the Internet, Global System for Mobile Communication (GSM), Wideband Code Division Multiple Access (WCDMA), 4G network, 5G network, Bluetooth, Wi-Fi, or other wireless or wired networks.
[0107] Example 4
[0108] This embodiment also provides a computer-readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, server, App application store, etc., which stores a computer program, and when the computer program is executed by a processor, it implements the steps of a table recognition method.
[0109] Obviously, those skilled in the art should understand that the modules or steps of the above-described embodiments of the present invention can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented here, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the embodiments of the present invention are not limited to any particular hardware and software combination.
[0110] It should be noted that the sequence numbers of the embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.
[0111] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.
[0112] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.
Claims
1. A method for table recognition, characterized in that, The method includes: Obtain an image to be identified, wherein the image to be identified includes tabular information and non-tabular information; The image to be identified is input into the trained table detection model to obtain the position information of the table region in the image to be identified. The image to be identified is then cropped using the position information of the table region to obtain the table image to be identified. The table image to be recognized is input into the trained table recognition model to obtain the pixel coordinates of the cell center region, the cell endpoint coordinates, and the offset distance feature map of the table image to be recognized. The offset distance feature map represents the offset relationship between the pixel coordinates of the cell center region and the cell endpoint coordinates. The position information of the cells in the table image to be identified is determined by the feature map of the pixel coordinates of the center region of the cells, the coordinates of the cell endpoints, and the offset distance of the cells in the table image to be identified. The position information of the cells in the table image to be recognized is input into the trained text recognition model to obtain the text information of the cells in the table image to be recognized. The table information corresponding to the table image to be recognized is generated by the position information of the cells in the table image to be recognized and the text information of the cells. The step of determining the position information of cells in the table image to be identified by using the pixel coordinates of the center region of the cell, the coordinates of the cell endpoints, and the offset distance feature map of the image to be identified includes: The cell coordinates corresponding to each central region pixel are determined by the pixel coordinates of the cell center region and the offset distance feature map; Calculate the distance between the cell coordinates corresponding to the pixel in the central region and the cell endpoint coordinates, filter the cell coordinates whose distance is less than or equal to a second preset threshold, and update the filtered cell coordinates using the cell endpoint coordinates whose distance is less than or equal to the second preset threshold; Determine whether the cell coordinates corresponding to the updated central region pixels match the cell endpoint coordinates; If so, the cell coordinates corresponding to the updated central region pixels are used as the cell position information in the table image to be identified; If not, extract the coordinates of points without matching results from the cell coordinates corresponding to the updated central region pixels, calculate the spatial distance between the point coordinates and the cell endpoint coordinates, update the point coordinates again using the endpoint with the smallest spatial distance from the point coordinates, and use the cell coordinates corresponding to the updated central region pixels as the position information of the cell in the table image to be identified.
2. The method according to claim 1, characterized in that, The step of inputting the table image to be recognized into the trained table recognition model to obtain the pixel coordinates of the cell center region, the cell endpoint coordinates, and the offset distance feature map of the table image to be recognized includes: The feature information of the table image to be identified is extracted by the downsampling module, and the feature information of the table image to be identified is decoded by the upsampling module to obtain the center region feature map, endpoint region feature map and offset distance feature map of the cell of the table image to be identified; Max pooling is performed on the central region feature map and the endpoint region feature map respectively to obtain the pooled central region feature map and the pooled endpoint region feature map. Obtain the same pixel values between the pooled center region feature map and the center region feature map, and the same pixel values between the pooled endpoint region feature map and the endpoint region feature map; Pixels with the same pixel value greater than a first preset threshold are respectively used as the pixel coordinates of the center region of the cell and the cell endpoint coordinates, and the cell endpoint coordinates are sorted according to a preset order.
3. The method according to claim 1, characterized in that, After determining the position information of cells in the table image to be identified through the pixel coordinates of the cell center region, the cell endpoint coordinates, and the offset distance feature map of the table image to be identified, the method further includes: The table border line clusters are determined based on the position information of the cells, wherein the table border line clusters include horizontal line clusters and vertical line clusters; The row and column index of the cell in the image to be identified is determined by the border line cluster of the table and the position information of the cell, wherein the row and column index includes the starting row index, the starting column index, the ending row index, and the ending column index.
4. The method according to claim 3, characterized in that, The step of determining the table border line cluster based on the cell position information includes: Calculate the first spatial distance between the top right corner coordinate of the cell and the top left corner coordinate of other cells, and the second spatial distance between the bottom right corner coordinate of the cell and the bottom left corner coordinate of other cells; Cells whose first spatial distance and second spatial distance are both less than a third preset threshold are generated into left and right adjacent cell clusters; Calculate the third spatial distance between the bottom left corner coordinate of the cell and the top left corner coordinate of other cells, and the fourth spatial distance between the bottom right corner coordinate of the cell and the top right corner coordinate of other cells; Cells whose third spatial distance and fourth spatial distance are both less than a third preset threshold are generated into vertically adjacent cell clusters; The first direction border lines of the left and right adjacent cell clusters are connected according to the order of the cells in the first direction. The connecting lines with a distance less than a third preset threshold are merged, and the merged border lines are sorted according to the second direction to generate the horizontal line clusters of the table. The second-direction border lines of the adjacent cell clusters are connected according to the order of the cells in the second direction. The connecting lines with a distance less than a third preset threshold are merged, and the merged border lines are sorted according to the first direction to generate the vertical line clusters of the table.
5. The method according to claim 3 or 4, characterized in that, Determining the row and column index of a cell in the image to be recognized by using the border line clusters of the table and the position information of the cell includes: Determine the feature points of the first direction border line of the cell, and calculate the distance between the feature points of the first direction border line and each horizontal line of the horizontal line cluster of the table. Select the index of the horizontal line with the smallest distance as the starting or ending row index of the cell; Determine the feature points of the second-direction border line of the cell, and calculate the distance between the feature points of the second-direction border line and each vertical line of the vertical line cluster of the table; Select the index of the vertical line with the smallest distance as the starting column index or ending column index of the cell.
6. A device for table recognition, characterized in that, The device includes: An acquisition module is used to acquire an image to be identified, wherein the image to be identified includes tabular information and non-tabular information; The cropping module is used to input the image to be recognized into the trained table detection model, obtain the position information of the table region in the image to be recognized, and crop the image to be recognized using the position information of the table region to obtain the table image to be recognized. The table recognition module is used to input the table image to be recognized into the trained table recognition model to obtain the pixel coordinates of the cell center region, the cell endpoint coordinates, and the offset distance feature map of the table image to be recognized, wherein the offset distance feature map represents the offset relationship between the pixel coordinates of the cell center region and the cell endpoint coordinates; The determination module is used to determine the position information of the cells in the table image to be identified by using the pixel coordinates of the center region of the cells, the cell endpoint coordinates, and the offset distance feature map of the table image to be identified. The text recognition module is used to input the table image to be recognized and the position information of the cells into the trained text recognition model to obtain the text information of the cells in the table image to be recognized, and to generate the table information corresponding to the table image to be recognized by using the position information of the cells in the table image to be recognized and the text information of the cells. Specifically, the determination module is used to: determine the cell coordinates corresponding to each central region pixel using the cell center region pixel coordinates and offset distance feature map; calculate the distance between the cell coordinates corresponding to the central region pixel and the cell endpoint coordinates, filter cell coordinates with a distance less than or equal to a second preset threshold, and update the filtered cell coordinates using the cell endpoint coordinates with a distance less than or equal to the second preset threshold; determine whether the updated cell coordinates corresponding to the central region pixel match the cell endpoint coordinates; if yes, use the updated cell coordinates corresponding to the central region pixel as the cell position information in the table image to be identified; if no, extract the coordinates of points without matching results from the updated cell coordinates corresponding to the central region pixel, calculate the spatial distance between the point coordinates and the cell endpoint coordinates, update the point coordinates again using the endpoint with the minimum spatial distance to the point coordinates, and use the updated cell coordinates corresponding to the central region pixel as the cell position information in the table image to be identified.
7. The apparatus according to claim 6, characterized in that, The table recognition module is specifically used for: The feature information of the table image to be identified is extracted by the downsampling module, and the feature information of the table image to be identified is decoded by the upsampling module to obtain the center region feature map, endpoint region feature map and offset distance feature map of the cell of the table image to be identified; Max pooling is performed on the central region feature map and the endpoint region feature map respectively to obtain the pooled central region feature map and the pooled endpoint region feature map. Obtain the same pixel values between the pooled center region feature map and the center region feature map, and the same pixel values between the pooled endpoint region feature map and the endpoint region feature map; The pixels with the same pixel value greater than the first preset threshold are respectively used as the pixel coordinates of the center area of the cell and the cell endpoint coordinates, and the cell endpoints are sorted according to a preset order.
8. A computer device, the computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor, when executing the computer program, implements the method according to any one of claims 1 to 5.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1 to 5.