Table recognition method, device, equipment, storage medium and program product

Abstract

The present application discloses a table recognition method, apparatus, device, storage medium and program product, which relates to the field of artificial intelligence technology, including: starting from a vertex cell of a target table in an image, each time the initial position information and layout information of a cell is recognized, the initial position information of the latest recognized cell is regularized based on the target position information of the recognized cells and the layout information of the latest recognized cells, and the target position information of the latest recognized cells is obtained, so that the latest recognized cells are aligned with other recognized cells; based on the target position information and layout information of the recognized cells, and the image of the target table area extracted from the image, the initial position information and layout information of the next cell are predicted. The present application improves the accuracy of the structure recognition results and ensures the alignment between the cells of the table reconstructed based on the structure recognition results.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a table recognition method, apparatus, device, storage medium, and program product. Background Art

[0002] Table recognition is an important branch of the document recognition field. It refers to the process of automatically detecting, parsing and extracting structured information in tables through computer technology, so that the table can be reconstructed based on the structured information.

[0003] When the structured information obtained by current table recognition methods is used to reconstruct a table, the resulting table may have overlapping cells. This requires complex post-processing of the structured information to reconstruct a table with aligned cells. This means that current table recognition methods have low recognition accuracy. Therefore, improving table recognition accuracy has become an urgent technical challenge. Summary of the Invention

[0004] In view of the above problems, this application provides a table recognition method, apparatus, device, storage medium and program product to improve the accuracy of table recognition. The specific solution is as follows:

[0005] A first aspect of the present application provides a table recognition method, comprising:

[0006] Performing structural recognition on a target table in the image to obtain a structural recognition result of the target table; the structural recognition result includes target position information and layout information of each cell in the target table; the layout information of each cell includes the number of rows and columns spanned by the cell;

[0007] Among them, structural recognition of the target table includes: starting from a vertex cell of the target table, each time the initial position information and layout information of a cell are recognized, the initial position information of the latest recognized cell is regularized based on the target position information of the recognized cell and the layout information of the latest recognized cell to obtain the target position information of the latest recognized cell, so that the latest recognized cell is aligned with other recognized cells; based on the target position information of the recognized cell and the image of the target table area extracted from the image, the initial position information and layout information of the next cell are predicted.

[0008] In a possible implementation, the method further includes:

[0009] Performing at least two structural recognitions on the target table in the image to obtain at least two structural recognition results; different structural recognitions start from different vertex cells of the target table;

[0010] Scoring each structure recognition result based on the image of the target table area;

[0011] The structure recognition result with the highest score is determined as the final structure recognition result of the target table.

[0012] In a possible implementation, scoring each structure recognition result based on the image of the target table area includes:

[0013] For each structure recognition result, a table reconstruction is performed according to the structure recognition result to obtain a reconstructed table image corresponding to the structure recognition result;

[0014] The reconstructed table image corresponding to the structure recognition result and the image of the target table area are input into a table scoring model to obtain a score for the structure recognition result.

[0015] In a possible implementation, the image includes at least one table, and the target table is any one of the at least one table; performing at least two structure recognitions on the target table in the image includes:

[0016] Performing table area recognition on the image to determine position information of each table in the image;

[0017] extracting an image of the target table area from the image according to the position information of the target table;

[0018] Performing vertex cell recognition on the image of the target table area to obtain initial position information and layout information of at least two vertex cells of the target table;

[0019] Corresponding to each identified vertex cell, the structure identification is performed once on the target table starting from the vertex cell.

[0020] In a possible implementation, the target table is the only table in the image; and performing at least two structure recognitions on the target table in the image includes:

[0021] Performing vertex cell recognition on the image to obtain initial position information and layout information of at least two vertex cells of the target table; the at least two vertex cells at least include two vertex cells on the same diagonal line of the target table;

[0022] Determining the position information of the target table according to the initial position information of the at least two vertex cells;

[0023] extracting an image of the target table area from the image according to the position information of the target table;

[0024] Corresponding to each identified vertex cell, the structure identification is performed once on the target table starting from the vertex cell.

[0025] In a possible implementation, vertex cell recognition is performed on the image to obtain initial position information and layout information of four vertex cells of the table;

[0026] The determining the position information of the target table according to the initial position information of the at least two vertex cells includes:

[0027] Extracting position information of four vertices of the target table from the initial position information of the four vertex cells;

[0028] The step of extracting the image of the target table area from the image according to the position information of the target table includes:

[0029] The area where the target table is located is intercepted from the image according to the position information of the four vertices and is rotationally corrected to obtain an image of the target table area.

[0030] In a possible implementation, performing vertex cell identification on the image includes:

[0031] The image is input into a vertex cell detection model to obtain initial position information and layout information of at least two vertex cells of the target table output by the vertex cell detection model.

[0032] In a possible implementation, predicting the initial position information and layout information of the next cell based on the target position information of the identified cell and the image of the target table area extracted from the image includes:

[0033] reconstructing an image of an identified cell according to target position information of the identified cell;

[0034] The identified cell image and the image of the target table area are input into a cell recognition model to obtain a cell recognition result output by the cell recognition model; the cell recognition result includes: whether there is a next cell, and the initial position information and layout information of the next cell if there is a next cell.

[0035] In a possible implementation, the regularizing of the initial position information of the newly recognized cell based on the target position information of the recognized cell and the layout information of the newly recognized cell includes:

[0036] Determine the neighbor cells of the most recently identified cell among the identified cells;

[0037] The initial position information of the newly identified cell is adjusted according to the layout information of the newly identified cell and the target position information of the neighbor cell, so that the size of the adjusted cell matches the layout information of the newly identified cell, and the adjusted cell is aligned with the adjacent edges of the neighbor cell.

[0038] In a possible implementation, the method further includes:

[0039] For each cell in the structure recognition result of the target table, intercepting an image of the cell area from the image of the target table area according to the target position information of the cell;

[0040] The image of the cell area is input into a text recognition model to obtain the text content in the cell.

[0041] A second aspect of the present application provides a table recognition device, comprising:

[0042] a structure recognition module, configured to perform structure recognition on a target table in the corresponding image and obtain a structure recognition result of the target table; the structure recognition result includes target position information and layout information of each cell in the target table; the layout information of each cell includes the number of rows and columns spanned by the cell;

[0043] Among them, structural recognition of the target table includes: starting from a vertex cell of the target table, each time the initial position information and layout information of a cell are recognized, the initial position information of the latest recognized cell is regularized based on the target position information of the recognized cell and the layout information of the latest recognized cell to obtain the target position information of the latest recognized cell, so that the latest recognized cell is aligned with other recognized cells; based on the target position information of the recognized cell and the image of the target table area extracted from the image, the initial position information and layout information of the next cell are predicted.

[0044] A third aspect of the present application provides a computer program product, comprising computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements the table recognition method of the first aspect or any implementation of the first aspect.

[0045] A fourth aspect of the present application provides an electronic device, comprising at least one processor and a memory connected to the processor, wherein:

[0046] The memory is used to store computer programs;

[0047] The processor is configured to execute the computer program so that the electronic device can implement the table recognition method of the first aspect or any implementation manner of the first aspect.

[0048] In a fifth aspect, the present application provides a computer storage medium carrying one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can use the table recognition method of the first aspect or any implementation of the first aspect.

[0049] By means of the above technical solution, the table recognition method, apparatus, device, storage medium and program product provided by the present application perform structural recognition on the target table in the image to obtain the structural recognition result of the target table; the structural recognition result includes the target position information and layout information of each cell in the target table; the layout information of each cell includes the number of rows and columns spanned by the cell; wherein, structural recognition of the target table includes: starting from a vertex cell of the target table, each time the initial position information and layout information of a cell are recognized, the initial position information of the latest recognized cell is regularized based on the target position information of the recognized cell and the layout information of the latest recognized cell to obtain the target position information of the latest recognized cell, so that the latest recognized cell is aligned with other recognized cells; based on the target position information and layout information of the recognized cell, the initial position information and layout information of the next cell are predicted. The table recognition method of the present application starts from the vertex cell and recognizes each cell one by one, and expands from the local to the whole. During the recognition process, each time the initial position information and layout information of a cell are recognized, the initial position information of the recognized cell is regularized to ensure that the recognized cells used to predict the next cell are aligned. Moreover, when recognizing the next cell, in addition to considering the local information of the recognized cell, the global information of the image of the target table area is also considered, which improves the accuracy of the structure recognition results and ensures that the cells of the table reconstructed based on the structure recognition results are aligned. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The above and other features, advantages, and aspects of the various embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that the originals and elements are not necessarily drawn to scale.

[0051] Figure 1 An example of a table with overlapping cells provided for this application;

[0052] Figure 2Examples of cell-aligned tables provided for this application;

[0053] Figure 3 A flowchart for implementing the method of regularizing the position information of the newly identified cells based on the position information of the identified cells and the layout information of the newly identified cells provided in the present application;

[0054] Figure 4 A flowchart for implementing the method of predicting the initial position information and layout information of the next cell based on the target position information of the identified cell and the image of the target table area extracted from the image provided by the present application;

[0055] Figure 5 An example of an image of the target table area provided for this application;

[0056] Figure 6 An example of an identified cell image provided for this application;

[0057] Figure 7 A flowchart of an implementation of the table recognition method provided in this application;

[0058] Figure 8 Another implementation flow chart of the table recognition method provided in this application;

[0059] Figure 9 This application is based on Figure 5 An example of a reconstructed table image obtained by performing table reconstruction on a structure recognition result corresponding to the target table shown;

[0060] Figure 10 A flowchart for implementing at least two structural recognitions of a target table in an image provided by the present application;

[0061] Figure 11 Another implementation flow chart of performing at least two structural recognitions on a target table in an image provided by the present application;

[0062] Figure 12 A schematic diagram of the structure of the table recognition device provided by this application;

[0063] Figure 13 A schematic diagram of the structure of the electronic device provided in this application. DETAILED DESCRIPTION

[0064] The following describes the embodiments of the present application in conjunction with the accompanying drawings. The terms used in the implementation methods of the present application are only used to explain the specific embodiments of the present application and are not intended to limit the present application.

[0065] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0066] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, and this is merely a way of distinguishing the objects of the same attributes when describing them in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0067] Tables, which organize information or data into rows and columns, are commonly used in communication, research, and data analysis. They appear in a variety of contexts, including books, papers, medical reports, financial analysis reports, and product manuals. The fundamental characteristics of tables are the representation of rows and columns and the organization of information. Tables organize data into a standard structure, making it easier to retrieve and compare information.

[0068] In some scenarios, documents exist in image format, such as scanning a paper document to generate a document image. When editing a document, text recognition is required to convert the image document into editable text. When an image document contains a table, it is also necessary to recognize the table in the image to obtain its structured information. This structured information can include the table's structural information (e.g., the location of each cell in the table) and the text within the table.

[0069] With the development of deep learning technology, table recognition methods based on deep learning have become the mainstream method for table recognition in academia and industry.

[0070] However, when the structured information obtained by the table recognition method based on deep learning technology is used for table reconstruction, the resulting table has overlapping cells. Figure 1 The figure below is an example of a table with overlapping cells provided by this application. In practical applications, users expect that the table restoration result (i.e., the reconstructed table) obtained based on the recognized structured information not only accurately restores the structure (the coordinates of the table itself, the rows and columns of the table, and the relationship between the rows and columns), but also accurately restores the positions of the cells within the table: the table lines between the cells must be aligned, and there must not be problems such as overlapping or disconnected cell coordinate boxes.

[0071] Currently, in order to align cells in a table, the identified structured information needs to be post-processed. This post-processing process needs to consider many situations, which makes the post-processing process more complicated and requires adding many rules to adapt to most situations. Figure 2 As shown, this is an example of a table with cell alignment provided in this application.

[0072] The inventors of this application have discovered that existing table recognition methods based on deep learning recognize tables from a holistic perspective, that is, they predict the coordinates of all cells in the entire table at one time. This table recognition method lacks local information, which may be the reason why the reconstructed table has overlapping cells.

[0073] In order to improve the accuracy of table recognition, this application scheme is proposed. The table recognition method provided by this application may include:

[0074] Perform structural recognition on the target table in the image to obtain the target table's structural recognition results. This structural recognition result includes the location information (referred to as target location information for ease of description and distinction) and layout information of each cell in the target table. The layout information of each cell includes the number of rows and columns that the cell spans.

[0075] Structural recognition of the target table refers to identifying the structural information of the target table, including but not limited to the position of each cell in the table within the image and the layout of each cell. Cell layout information represents the merging of cells. For example, if a cell's layout information indicates that it spans n rows and m columns, then the cell is formed by merging n×m minimum cells in m consecutive columns from the minimum cells in n consecutive rows. Both n and m are positive integers greater than 0.

[0076] Optionally, the process of performing structure recognition on the target table may include:

[0077] Starting from a vertex cell of the target table, each time the initial position information and layout information of a cell are identified, the initial position information of the latest identified cell is regularized based on the target position information of the identified (i.e., historically identified) cells and the layout information of the latest identified cell to obtain the target position information of the latest identified cell, so that the latest identified cell is aligned with other identified cells.

[0078] Vertex cells are cells at the vertices of a table. Table 1 shows an example of a table provided in an embodiment of the present application. In this example, the number in each cell is the cell number, where cell 1, cell 6, cell 21, and cell 27 are all vertex cells.

[0079] Table 1

[0080]

[0081] Cell No. 1 spans one row and two columns, and the other cells span one row and one column.

[0082] For each structure recognition, the initial position information of the first recognized cell (i.e., the vertex cell) does not need to be regularized. Instead, the initial position of the vertex cell is used as the alignment reference for subsequently recognized cells, and the initial position information of the vertex cell is directly used as the target position information of the vertex cell.

[0083] Alternatively, for each structure recognition, the image can be used as an alignment reference, with the goal of aligning the edge of the first recognized cell with the edge of the image, and the initial position information of the first recognized cell can be regularized to obtain the target position information of the first recognized cell.

[0084] In an optional embodiment, a flowchart for implementing the above-mentioned regularization of the position information of the newly identified cells based on the position information of the identified cells and the layout information of the newly identified cells is as follows: Figure 3 As shown, this may include:

[0085] Step S301: Determine neighbor cells of the most recently identified cell among the identified cells.

[0086] When performing structural recognition on a target table, cells are usually recognized one by one in a preset order, for example, in a row-by-row and column-by-column order. Taking Table 1 as an example, assuming that we start from cell 1, after recognizing cell 1, we recognize cell 2, then cell 3, and so on. After recognizing cell 6, we recognize cell 7, then cell 8, and so on, until we recognize cell 27. For another example, assuming that we start from cell 6, after recognizing cell 6, we recognize cell 5, then cell 4, and so on. After recognizing cell 1, we recognize cell 13, then cell 12, and so on, until we recognize cell 21. For another example, assuming that we start from cell 27, after recognizing cell 27, we recognize cell 26, then cell 25, and so on. After recognizing cell 21, we recognize cell 20, then cell 19, and so on, until we recognize cell 1.

[0087] Assuming the most recently recognized cell is the i-th cell, we can first determine whether the left and right boundaries of the last row of cells in the first i-1 recognized cells are aligned with the left and right boundaries of the cells in the previous row. If so, the i-th cell's neighbor cells include at least the first cell in the last row of cells in the first i-1 recognized cells. If the last row of cells in the first i-1 recognized cells is aligned only with the left edge of the table (or only with the right edge of the table), the i-th cell's neighbor cells include the i-1-th cell. If the i-1-th cell is not a cell in the first row, the i-th cell's neighbor cells also include at least one cell in the row above the i-1-th cell. i is a positive integer greater than 1, i.e., i=2, 3, 4, ...

[0088] Step S302: adjusting the position information of the newly identified cell according to the layout information of the newly identified cell and the target position information of the neighbor cell, so that the size of the adjusted cell matches the layout information of the newly identified cell and the adjacent edges of the adjusted cell and the neighbor cell are aligned.

[0089] Optionally, the vertex coordinates of the newly recognized cell can be adjusted with the goal of aligning the adjacent edges of the newly recognized cell with those of its neighbor cells (i.e., the adjacent edges of the newly recognized cell and its neighbor cells overlap) so that the adjacent edges of the newly recognized cell and its neighbor cells are aligned. Then, with the goal of matching the size of the newly recognized cell with its layout information, the vertex coordinates of the newly recognized cell can be continued to be adjusted so that the size of the adjusted cell matches the layout information of the newly recognized cell, and the adjacent edges of the adjusted cell and the neighbor cells are aligned.

[0090] The initial position information and layout information of the next cell are predicted based on the target position information of the identified cell and the image of the target table area extracted from the image.

[0091] The target table's location information can be first identified from the image, and then an image of the target table area can be extracted from the image based on the target table's location information. The target table's location information can be the coordinates of any two diagonal vertices of the target table, or the target table's location information can be the coordinates of four vertices of the target table.

[0092] Optionally, a flowchart for implementing the above-mentioned prediction of the initial position information and layout information of the next cell based on the target position information of the identified cell and the image of the target table area extracted from the image is as follows: Figure 4 As shown, this may include:

[0093] Step S401: reconstructing the recognized cell image according to the target position information of the recognized cell.

[0094] The reconstructed recognized cell image contains all recognized cells with target location information, and there is no text content in the cells. Figure 5 As shown in FIG, an example of an image of the target table area provided in an embodiment of the present application is provided. Figure 6 The figure shows an example of an identified cell image provided by an embodiment of the present application. Figure 6 In the example shown, 8 cells have been identified. Figure 6 It is an image of the identified cells reconstructed based on the target position information of the 8 identified cells.

[0095] Step S402: Input the identified cell image and the image of the target table area into the cell recognition model to obtain the cell recognition result output by the cell recognition model; the cell recognition result includes: whether there is a next cell, and the initial position information and layout information of the next cell if there is a next cell.

[0096] by Figure 6 For example, when predicting the initial position information and layout information of the 9th cell, Figure 6 The identified cell images and Figure 5 The image of the target table area shown is input into the cell recognition model to obtain the recognition result corresponding to the 9th cell output by the cell recognition model. The recognition result includes: whether there is a 9th cell, and when there is a 9th cell, the initial position information and layout information of the 9th cell.

[0097] Optionally, the cell recognition model is trained using image pairs as training samples and the cell recognition results corresponding to the image pairs as labels. The image pairs include an image of the table to be recognized and an image of an identified cell reconstructed based on cells recognized in the image of the table to be recognized. The cell recognition results corresponding to the image pairs indicate whether there is a next cell in the image of the table to be recognized, as well as the location and layout information of the next cell if there is one.

[0098] Optionally, a flowchart of an implementation of the table recognition method provided in the embodiment of the present application is as follows: Figure 7 As shown, this may include:

[0099] Step S701: using the target position information of the first i-1 cells and the image of the target table area, predict the initial position information and layout information of the i-th cell.

[0100] That is to say, after obtaining the target position information and layout information of the first cell, the initial position information and layout information of the second cell are obtained based on the target position information of the first cell and the image prediction of the target table area; the initial position information and layout information of the third cell are obtained based on the target position information of the first cell, the target position information of the second cell and the image prediction of the target table area; the initial position information and layout information of the fourth cell are obtained based on the target position information of the first cell, the target position information of the second cell, the target position information of the third cell and the image prediction of the target table area, and so on, until all cells are identified.

[0101] The target position information and layout information of the first cell (i.e., the vertex cell) can be obtained by directly processing the image, or by regularizing the initial position information of the first cell based on the image. For details, please refer to the above embodiment.

[0102] Step S702: regularize the initial position information of the ith cell based on the target position information of the first i-1 cells and the layout information of the ith cell to obtain the target position information of the ith cell, so that the ith cell is aligned with the first i-1 cells.

[0103] Since the first i-1 cells are aligned (ie, there is no cell overlap), it is only necessary to align the position of the i-th cell with the identified cell, which is simple to implement. The specific regularization process can be found in the above embodiment and will not be repeated here.

[0104] Step S703: add 1 to the value of i. Return to step S701 and continue until all cells are identified (i.e., the prediction result indicates that there is no next cell).

[0105] In an optional embodiment, another implementation flow chart of the table recognition method provided by the present application is as follows: Figure 8 As shown, this may include:

[0106] Step S801: Perform at least two structural recognitions on the target table in the image to obtain at least two structural recognition results. Different structural recognitions start from different vertex cells of the target table.

[0107] That is, different structural recognition results of the target table are obtained by performing different sub-structure recognitions on the target table starting from different vertex cells of the target table.

[0108] Each structure recognition process can refer to the above embodiments and will not be described in detail here.

[0109] Step S802: scoring each structure recognition result based on the image of the target table area to obtain a score for each structure recognition result.

[0110] The score of each structure recognition result represents the degree of restoration of the structure recognition result to the target table. The higher the score, the higher the degree of restoration of the structure recognition result to the target table. As an example, the score of the structure recognition result is between 0 and 1.

[0111] Optionally, you can score each structure recognition result as follows:

[0112] For each structure recognition result, a table reconstruction is performed according to the structure recognition result to obtain a reconstructed table image corresponding to the structure recognition result.

[0113] As an example, a table in image format may be reconstructed according to the target position information of each cell in the structure recognition result (referred to as reconstructed table image for ease of description and distinction).

[0114] like Figure 9 As shown, the embodiment of the present application provides Figure 5 An example of a reconstructed table image obtained by reconstructing a table using a structure recognition result corresponding to a target table (the structure recognition result is obtained based on the structure recognition method of this application). Obviously, the reconstructed table image only contains the table and does not contain the text content in the target table.

[0115] The reconstructed table image corresponding to the structure recognition result and the image of the target table area are input into the table scoring model to obtain the score of the structure recognition result.

[0116] Step S803: Determine the structure recognition result with the highest score as the final structure recognition result of the target table.

[0117] This application further improves the accuracy of table recognition by performing at least two structural recognitions on the target table, scoring each structural recognition result, and using the structural recognition result with the highest score as the final structural recognition result of the target table.

[0118] In an optional embodiment, the image includes at least one table, that is, the image may contain only one table (i.e., the target table), or may contain multiple tables, and the target table is any one of the multiple tables. In the case where the image includes multiple tables, the recognition of each table is independent of each other, and the recognition method of different tables is the same. Based on this, a flowchart for implementing the above-mentioned at least two structural recognitions of the target table in the image is as follows: Figure 10 As shown, this may include:

[0119] Step S1001: performing table area recognition on the image to determine the position information of each table in the image.

[0120] Optionally, a table area recognition model may be used to perform table area recognition on an image. As an example, an image may be input into a pre-trained table area recognition model to obtain position information of each table output by the table area recognition model.

[0121] The position information of each table includes the coordinates of two diagonally opposite vertices of the table.

[0122] Alternatively, the position information of each table includes the coordinates of four vertices of the table.

[0123] The table area recognition model is trained by using images containing at least one table as training samples and the position information of each table in the image as labels.

[0124] The table area recognition model can be a small model or a large model.

[0125] Step S1002: extracting an image of the target table area from the image according to the location information of the target table.

[0126] If the target table's location information is the coordinates of two diagonal vertices, a rectangular region can be determined in the image based on the coordinates of the two vertices. This region can then be captured from the image to obtain an image of the target table's region. This approach can extract an image of the entire target table's region if the target table is not deformed. However, if the target table's rows are deformed, the entire target table's region may not be captured.

[0127] When the location information of the target table includes the coordinates of the four vertices of the target table, the four vertex coordinates can be connected to obtain a quadrilateral, and the quadrilateral area (that is, the area where the target table is located) can be cut out from the image to obtain the initial table area image, and the initial table area image can be rotationally corrected to obtain the image of the target table area.

[0128] As an example, the angle of the target form (denoted as θ) can be calculated based on the angle between two adjacent edges of the quadrilateral and one edge of the image. This angle is the angle resulting from the geometric or perspective transformation of the target form relative to the frontal vertical image due to the shooting angle. The specific calculation process can be found in existing solutions and will not be detailed here.

[0129] Calculate the transformation matrix (denoted as M) based on the angle θ of the target table.

[0130] Specifically, we can first calculate the rotation matrix R when the rotation center is the origin (0, 0) based on the angle θ of the target table:

[0131] .

[0132] If you want to use the center of the table (cx, cy) as the rotation center, you need to translate the center of the table to the origin. The transformation matrix T1 is:

[0133] .

[0134] Then translate the coordinate system back to the original rotation center, and the transformation matrix T2 is:

[0135] .

[0136] Based on this, M= .

[0137] The process of using transformation matrix M to perform rotation correction on the initial table area image involves applying transformation matrix M to all coordinates in the initial table area image, then setting the pixel values ​​of the transformed coordinates to the pixel values ​​of the corresponding coordinates in the initial table area image before the transformation. This can be achieved using the OpenCV warpAffine function. By obtaining the coordinates of the four vertices of the target table, extracting the initial table area image from the image based on these four vertex coordinates, and performing rotation correction on the initial table area image, the entire table area can be obtained as much as possible.

[0138] Step S1003: performing vertex cell recognition on the image of the target table area to obtain initial position information and layout information of at least two vertex cells of the target table.

[0139] Optionally, the image of the target table area may be input into a vertex cell detection model to obtain position information and layout information of at least two vertex cells of the target table output by the vertex cell detection model.

[0140] When the initial position information and layout information of two vertex cells of the target table are obtained, the two vertex cells can be two vertex cells located in the same row of the target table, or can be two vertex cells located in the same column of the target table, or can be two vertex cells belonging to opposite corners of the target table.

[0141] Step S1004: For each identified vertex cell, a structure recognition is performed on the target table starting from the vertex cell.

[0142] The specific structure recognition process can be found in the above embodiment and will not be described in detail here.

[0143] In an optional embodiment, when the target table is the only table in the image, the image can be an image input by the user, or an image with a unique table captured from the image input by the user. Figure 11 As shown, this may include:

[0144] Step S1101: performing vertex cell recognition on the image to obtain initial position information and layout information of at least two vertex cells of the target table; the at least two vertex cells include at least two vertex cells on the same diagonal line of the target table.

[0145] Optionally, vertex cell recognition may be performed on the image to obtain initial position information and layout information of two vertex cells of the target table; the two vertex cells are two vertex cells containing diagonal vertices of the target table.

[0146] Optionally, vertex cell recognition may be performed on the image to obtain initial position information and layout information of three vertex cells of the target table.

[0147] Optionally, vertex cell recognition may be performed on the image to obtain initial position information and layout information of four vertex cells of the target table.

[0148] Optionally, the image may be input into a vertex cell detection model to obtain initial position information and layout information of at least two vertex cells of a target table output by the vertex cell detection model.

[0149] The vertex cell detection model can be trained using images containing a unique table as training samples, and using the position information and layout information of at least two vertex cells corresponding to the image as labels. Labels corresponding to different training samples may contain position information and layout information of vertex cells in the same position (e.g., both the top left vertex cell and the bottom right vertex cell of the target table; or both the top right vertex cell and the bottom left vertex cell of the target table; or all four vertex cells of the target table).

[0150] In some cases, such as Figure 5 As shown, among the four identified vertex cells, there may be two vertex cells whose initial position information and layout information are the same, for example, the upper left vertex cell and the upper right vertex cell.

[0151] The position information of each vertex cell includes the coordinates of the four vertices of the vertex cell, or the position information of each vertex cell may include the coordinates of the two diagonal vertices of the vertex cell, or the position information of each vertex cell may include the coordinates of the three vertices of the vertex cell. In the case where the position information of each vertex cell includes the coordinates of two or three vertices of the vertex cell, the position information of the upper left vertex cell and the lower right vertex cell includes at least the upper left vertex and the lower right vertex of the vertex cell, and the position information of the upper right vertex cell and the lower left vertex cell includes at least the upper right vertex and the lower left vertex of the vertex cell.

[0152] Step S1102: determining the position information of the target table according to the initial position information of at least two vertex cells.

[0153] Optionally, the coordinates of two diagonally opposite vertices of the target table may be extracted from the initial position information of at least two vertex cells as the position information of the target table.

[0154] Optionally, the coordinates of the four vertices of the target table may be extracted from the initial position information of the four vertex cells as the position information of the target table.

[0155] Step S1103: extracting an image of the target table area from the image according to the location information of the target table.

[0156] If the target table's location information is the coordinates of two diagonal vertices, a rectangular region can be determined based on the coordinates of the two vertices. This region can then be captured from the image to obtain an image of the target table region. This approach can accurately extract an image of the target table region if the target table is not deformed. However, if the target table has rows deformed, the complete table region may not be captured.

[0157] When the target table's location information is the coordinates of the target table's four vertices, the target table's region can be captured from the image based on the four vertex location information and rotationally corrected to obtain an image of the target table's region. Specifically, the four vertex coordinates can be connected to form a quadrilateral, and the quadrilateral region (i.e., the target table's region) can be captured from the image to obtain an initial image of the region. This initial image of the region can then be rotationally corrected to obtain an image of the target table's region. The specific implementation method can be found in the aforementioned embodiments and will not be further elaborated here.

[0158] Step S1104: For each identified vertex cell, perform a structure recognition on the target table starting from the vertex cell.

[0159] The specific structure recognition process can be found in the above embodiment and will not be described in detail here.

[0160] Furthermore, for each identified vertex cell, the initial position information of the vertex cell can be rotationally corrected using the aforementioned transformation matrix M. When performing structure recognition on the target table, the target position information of the vertex cell is the initial position information after rotation correction.

[0161] For any coordinate in the initial position information of the vertex cell, denoted as (x, y), the homogeneous coordinate of any coordinate is (x, y, 1). The transformation matrix M can be used to align the secondary coordinates for rotation correction, and the homogeneous coordinates after rotation correction are (x', y', w'). Then, normalization is performed to obtain the coordinates after rotation correction corresponding to the coordinates (x, y). . It can be expressed as:

[0162] .

[0163] In an optional embodiment, the table recognition method provided by the present application may further include:

[0164] For each cell in the structure recognition result of the target table, an image of the cell area is intercepted from the image of the target table area according to the target position information of the cell.

[0165] The captured image of the cell area is input into the text recognition model to obtain the text content in the cell.

[0166] That is to say, in addition to the cell-by-cell recognition of the structure of the target table, the text content in the target table is also recognized cell-by-cell.

[0167] Optionally, text recognition can be performed on each cell every time the target position information of a cell is identified. In this way, when the target table in the image is subjected to at least two structural recognitions, at least two text recognitions need to be performed for the same cell, which consumes more computing resources. In order to reduce the consumption of computing resources, when the target table in the image is subjected to at least two structural recognitions, after determining the final structural recognition result, the image of the cell area can be intercepted from the image of the target table area based on the target position information of each cell in the final structural recognition result, and the intercepted image of the cell area can be input into the text recognition model to obtain the text content in the cell.

[0168] Corresponding to the method embodiment, the present application also provides a table recognition device. A structural diagram of the table recognition device provided in the embodiment of the present application is shown as follows: Figure 12 As shown, this may include:

[0169] The structure recognition module 1201 is configured to perform structure recognition on a target table in an image to obtain a structure recognition result of the target table; the structure recognition result includes target position information and layout information of each cell in the target table; the layout information of each cell includes the number of rows and columns spanned by the cell;

[0170] Among them, structural recognition of the target table includes: starting from a vertex cell of the target table, each time the initial position information and layout information of a cell are recognized, the initial position information of the latest recognized cell is regularized based on the target position information of the recognized cell and the layout information of the latest recognized cell to obtain the target position information of the latest recognized cell, so that the latest recognized cell is aligned with other recognized cells; based on the target position information of the recognized cell and the image of the target table area extracted from the image, the initial position information and layout information of the next cell are predicted.

[0171] The table recognition device provided in the embodiment of the present application starts from the vertex cell and recognizes each cell one by one, and expands from the local to the whole. During the recognition process, each time the initial position information and layout information of a cell are recognized, the initial position information of the recognized cell is regularized to ensure that the recognized cells used to predict the next cell are aligned. Moreover, when recognizing the next cell, in addition to considering the local information of the recognized cell, the global information of the image of the target table area is also considered, thereby improving the accuracy of the structure recognition result and ensuring the alignment between the cells of the table reconstructed based on the structure recognition result.

[0172] In an optional embodiment, the table recognition device provided by the present application may further include:

[0173] a control module, configured to perform at least two structural recognitions on the target table in the image to obtain at least two structural recognition results; wherein different structural recognitions are started from different vertex cells of the target table;

[0174] A scoring module, configured to score each structure recognition result based on the image of the target table area;

[0175] The determination module is configured to determine the structure recognition result with the highest score as the final structure recognition result of the target table.

[0176] In an optional embodiment, when the scoring module scores each structure recognition result based on the image of the target table area, it is configured to:

[0177] For each structure recognition result, a table reconstruction is performed according to the structure recognition result to obtain a reconstructed table image corresponding to the structure recognition result;

[0178] The reconstructed table image corresponding to the structure recognition result and the image of the target table area are input into a table scoring model to obtain a score for the structure recognition result.

[0179] In an optional embodiment, the image includes at least one table, and the target table is any one of the at least one table; when the control module performs at least two structure recognitions on the target table in the image, it is configured to:

[0180] Performing table area recognition on the image to determine position information of each table in the image;

[0181] extracting an image of the target table area from the image according to the position information of the target table;

[0182] Performing vertex cell recognition on the image of the target table area to obtain initial position information and layout information of at least two vertex cells of the target table;

[0183] Corresponding to each identified vertex cell, the structure identification module 1201 is called to perform the structure identification on the target table starting from the vertex cell.

[0184] In an optional embodiment, the target table is the only table in the image; when the control module performs at least two structural recognitions on the target table in the image, it is configured to:

[0185] Performing vertex cell recognition on the image to obtain initial position information and layout information of at least two vertex cells of the target table; the at least two vertex cells at least include two vertex cells on the same diagonal line of the target table;

[0186] Determining the position information of the target table according to the initial position information of the at least two vertex cells;

[0187] extracting an image of the target table area from the image according to the position information of the target table;

[0188] Corresponding to each identified vertex cell, the structure identification module 1201 is called to execute the structure identification process once on the target table starting from the vertex cell.

[0189] In an optional embodiment, the control module performs vertex cell recognition on the image to obtain initial position information and layout information of four vertex cells of the table;

[0190] When the control module determines the position information of the target table according to the initial position information of the at least two vertex cells, it is configured to:

[0191] Extracting position information of four vertices of the target table from the initial position information of the four vertex cells;

[0192] When the control module extracts the image of the target table area from the image according to the position information of the target table, it is used to:

[0193] The area where the target table is located is intercepted from the image according to the position information of the four vertices and is rotationally corrected to obtain an image of the target table area.

[0194] In an optional embodiment, when the control module performs vertex cell identification on the image, it is configured to:

[0195] The image is input into a vertex cell detection model to obtain initial position information and layout information of at least two vertex cells of the target table output by the vertex cell detection model.

[0196] In an optional embodiment, when predicting the initial position information and layout information of the next cell based on the target position information of the identified cell and the image of the target table area extracted from the image, the structure recognition module 1201 is configured to:

[0197] reconstructing an image of an identified cell according to target position information of the identified cell;

[0198] The identified cell image and the image of the target table area are input into a cell recognition model to obtain a cell recognition result output by the cell recognition model; the cell recognition result includes: whether there is a next cell, and the initial position information and layout information of the next cell if there is a next cell.

[0199] In an optional embodiment, when the structure recognition module 1201 regularizes the initial position information of the newly recognized cell based on the target position information of the recognized cell and the layout information of the newly recognized cell, it is configured to:

[0200] Determine the neighbor cells of the most recently identified cell among the identified cells;

[0201] The initial position information of the newly identified cell is adjusted according to the layout information of the newly identified cell and the target position information of the neighbor cell, so that the size of the adjusted cell matches the layout information of the newly identified cell, and the adjusted cell is aligned with the adjacent edges of the neighbor cell.

[0202] In an optional embodiment, the table recognition device further includes:

[0203] The text recognition module is used to capture an image of the cell area from the image of the target table area for each cell in the structural recognition result of the target table according to the target position information of the cell; input the image of the cell area into the text recognition model to obtain the text content in the cell.

[0204] An electronic device is also provided in an embodiment of the present application. Figure 13 , which shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiments of the present application. The electronic device in the embodiments of the present application can be a terminal device (such as a car computer, a large-screen device, a smart home, a mobile phone, a tablet computer, a laptop computer, a desktop computer, etc.), or a server (which can be a single server, a server cluster, or a cloud server, etc.). Figure 13 The electronic device shown is merely an example and should not limit the functions and scope of use of the embodiments of the present application.

[0205] like Figure 13 As shown, the electronic device may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 1301, which can perform various appropriate actions and processes based on programs stored in a read-only memory (ROM) 1302 or programs loaded from a storage device 1308 into a random access memory (RAM) 1303. When the electronic device is powered on, the RAM 1303 also stores various programs and data required for the operation of the electronic device. The processing device 1301, ROM 1302, and RAM 1303 are interconnected via a bus 1304. An input / output (I / O) interface 1305 is also connected to the bus 1304.

[0206] Typically, the following devices may be connected to the I / O interface 1305: an input device 1306 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 1307 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 1308 including, for example, a memory card, a hard disk, etc.; and a communication device 1309. The communication device 1309 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Figure 13The electronic device is shown with various devices, but it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed instead.

[0207] An embodiment of the present application also provides a computer program product including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements any one of the table recognition methods provided in the embodiments of the present application.

[0208] A computer-readable storage medium is also provided in an embodiment of the present application. The storage medium carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any table recognition method provided in the embodiment of the present application.

[0209] It should be noted that the device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed across multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment. In addition, in the drawings of the device embodiments provided in this application, the connection relationship between the modules indicates that there is a communication connection between them, which can be specifically implemented as one or more communication buses or signal lines.

[0210] Through the description of the above embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software plus necessary general-purpose hardware, and of course can also be implemented by dedicated hardware including application-specific integrated circuits, dedicated CPUs, dedicated memories, dedicated components, etc. In general, all functions performed by computer programs can be easily implemented with corresponding hardware, and the specific hardware structures used to implement the same function can also be diverse, such as analog circuits, digital circuits, or dedicated circuits. However, for the present application, software program implementation is a better implementation method in most cases. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer's floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk or optical disk, etc., and includes a number of instructions to enable a computer device (which can be a personal computer, training equipment, or network equipment, etc.) to execute the methods described in each embodiment of the present application.

[0211] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments may be implemented in the form of a computer program product. Professionals and technicians may use different methods to implement the described functions for each specific solution, but such implementation should not be considered beyond the scope of this application.

[0212] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website, a computer, a training device or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website, a computer, a training device or a data center. The computer-readable storage medium can be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium can be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

[0213] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0214] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present application. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application is not limited to the embodiments shown herein, but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A table recognition method, characterized in that: include: Performing structural recognition on a target table in the image in a cell-by-cell recognition manner to obtain a structural recognition result of the target table; The structure recognition result includes target position information and layout information of each cell in the target table; the layout information of each cell includes the number of rows and columns spanned by the cell; Among them, structural recognition of the target table includes: starting from a vertex cell of the target table, each time the initial position information and layout information of a cell are recognized, the initial position information of the latest recognized cell is regularized based on the target position information of the recognized cell and the layout information of the latest recognized cell to obtain the target position information of the latest recognized cell, so that the latest recognized cell is aligned with other recognized cells; based on the target position information of the recognized cell and the image of the target table area extracted from the image, the initial position information and layout information of the next cell are predicted.

2. The method according to claim 1, characterized in that Also includes: Performing at least two structural recognition operations on the target table in the image to obtain at least two structural recognition results; Different substructure identifications start from different vertex cells of the target table; Scoring each structure recognition result based on the image of the target table area; The structure recognition result with the highest score is determined as the final structure recognition result of the target table.

3. The method according to claim 2, characterized in that Scoring each structure recognition result based on the image of the target table area includes: For each structure recognition result, a table reconstruction is performed according to the structure recognition result to obtain a reconstructed table image corresponding to the structure recognition result; The reconstructed table image corresponding to the structure recognition result and the image of the target table area are input into a table scoring model to obtain a score for the structure recognition result.

4. The method according to claim 2, characterized in that The image includes at least one table, and the target table is any one of the at least one table; performing at least two structure recognitions on the target table in the image includes: Performing table area recognition on the image to determine position information of each table in the image; extracting an image of the target table area from the image according to the position information of the target table; Performing vertex cell recognition on the image of the target table area to obtain initial position information and layout information of at least two vertex cells of the target table; Corresponding to each identified vertex cell, the structure identification is performed once on the target table starting from the vertex cell.

5. The method according to claim 2, characterized in that The target table is the only table in the image; The performing at least two structural recognitions on the target table in the image includes: Performing vertex cell recognition on the image to obtain initial position information and layout information of at least two vertex cells of the target table; the at least two vertex cells at least include two vertex cells on the same diagonal line of the target table; Determining the position information of the target table according to the initial position information of the at least two vertex cells; extracting an image of the target table area from the image according to the position information of the target table; Corresponding to each identified vertex cell, the structure identification is performed once on the target table starting from the vertex cell.

6. The method according to claim 5, characterized in that Performing vertex cell recognition on the image to obtain initial position information and layout information of four vertex cells of the table; The determining the position information of the target table according to the initial position information of the at least two vertex cells includes: Extracting position information of four vertices of the target table from the initial position information of the four vertex cells; The step of extracting the image of the target table area from the image according to the position information of the target table includes: The area where the target table is located is intercepted from the image according to the position information of the four vertices and is rotationally corrected to obtain an image of the target table area.

7. The method according to claim 5, characterized in that Performing vertex cell identification on the image includes: The image is input into a vertex cell detection model to obtain initial position information and layout information of at least two vertex cells of the target table output by the vertex cell detection model.

8. The method according to claim 1, characterized in that The method of predicting the initial position information and layout information of the next cell based on the target position information of the identified cell and the image of the target table area extracted from the image includes: reconstructing an image of an identified cell according to target position information of the identified cell; The identified cell image and the image of the target table area are input into a cell recognition model to obtain a cell recognition result output by the cell recognition model; the cell recognition result includes: whether there is a next cell, and the initial position information and layout information of the next cell if there is a next cell.

9. The method according to claim 1, characterized in that The regularizing of the initial position information of the newly identified cell based on the target position information of the identified cell and the layout information of the newly identified cell includes: Determine the neighbor cells of the most recently identified cell among the identified cells; The initial position information of the newly identified cell is adjusted according to the layout information of the newly identified cell and the target position information of the neighbor cell, so that the size of the adjusted cell matches the layout information of the newly identified cell, and the adjusted cell is aligned with the adjacent edges of the neighbor cell.

10. The method according to any one of claims 1 to 9, characterized in that Also includes: For each cell in the structure recognition result of the target table, intercepting an image of the cell area from the image of the target table area according to the target position information of the cell; The image of the cell area is input into a text recognition model to obtain the text content in the cell.

11. A table recognition device, characterized in that: include: A structure recognition module is used to perform structure recognition on a target table in an image in a cell-by-cell manner to obtain a structure recognition result of the target table; The structure recognition result includes target position information and layout information of each cell in the target table; the layout information of each cell includes the number of rows and columns spanned by the cell; Among them, structural recognition of the target table includes: starting from a vertex cell of the target table, each time the initial position information and layout information of a cell are recognized, the initial position information of the latest recognized cell is regularized based on the target position information of the recognized cell and the layout information of the latest recognized cell to obtain the target position information of the latest recognized cell, so that the latest recognized cell is aligned with other recognized cells; based on the target position information of the recognized cell and the image of the target table area extracted from the image, the initial position information and layout information of the next cell are predicted.

12. A computer program product, characterized in that The method comprises computer-readable instructions, which, when executed on an electronic device, enable the electronic device to implement the table recognition method according to any one of claims 1 to 10.

13. An electronic device, characterized in that: The electronic device comprises at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is configured to execute the computer program so as to enable the electronic device to implement the table recognition method according to any one of claims 1 to 10.

14. A computer storage medium, characterized in that The storage medium carries one or more computer programs, and when the one or more computer programs are executed by an electronic device, the electronic device can implement the table recognition method according to any one of claims 1 to 10.

Images