Table recognition method and device

Abstract

The present invention discloses a table recognition method, which obtains an image to be processed that includes a table, and determines information of each cell in the image to be processed, where the information of each cell includes the position of the perimeter of each cell. Then, based on the information of each cell, obtain a parent cell in the row direction of each cell and a parent cell in the column direction of each cell. Furthermore, based on the parent-child relationship of each cell in the row direction and the parent-child relationship of each cell in the column direction, obtain a structural coordinate of each cell, where the structural coordinate includes a start row, a start column, an end row, and an end column. The parent-child relationship of cells in the row direction and the parent-child relationship of cells in the column direction are simpler than a graph structure. Therefore, the present solution can reduce the amount of calculation for determining the structural coordinate of a cell.

Description

[Technical field]

[0001] Cross-references to related applications This application claims priority to a Chinese patent application filed on January 7, 2022 with the State Intellectual Property Office of the People's Republic of China, application number 202210018252.1, and invention title "Table recognition method and apparatus", the entire contents of which are incorporated herein by reference. Technical Field The present invention relates to the field of image processing, and in particular to a table recognition method and apparatus. [Background technology]

[0002] Table recognition consists of two parts: Table Structure Recognition (TSR) and Table Content Recognition (Table The method includes two parts: structure recognition and content recognition (TCR). Here, structure recognition refers to analyzing the row and column positions of each cell and the physical position of the specific box from an image containing a table. Content recognition refers to recognizing the text in each cell. Through the two steps of structure recognition and content recognition, the structure coordinates of each cell and the text content in each cell can be obtained, and the table can be converted into formats such as Excel or Word, which facilitates further manual proofreading and editing and greatly facilitates the digitization process of table information.

[0003] Currently, methods for recognizing table structures are more complicated and tend to consume more computational resources.

[0004] Therefore, there is an urgent need for a solution that can solve the above problems. Summary of the Invention

[0005] The technical problem to be solved by the present invention is how to easily recognize a table structure, and a method and device for recognizing a table are provided.

[0006] According to a first aspect, an embodiment of the present invention provides a table recognition method, the method comprising: obtaining a ready image including a table and determining information for each cell in the ready image, the information for each cell including a location of a bounding box for each cell; obtaining a parent cell in a row direction of each of the cells and a parent cell in a column direction of each of the cells based on information of each of the cells; obtaining structural coordinates of each of the cells based on a parent-child relationship in the row direction of each of the cells and a parent-child relationship in the column direction of each of the cells, the structural coordinates including a start row, a start column, an end row, and an end column.

[0007] Optionally, the information for each cell comprises: The data further includes a word embedding vector of the text in each of the cells and / or visual features of each of the cells.

[0008] Optionally, obtaining a parent cell in a row direction of each of the cells and a parent cell in a column direction of each of the cells based on information of each of the cells includes: The method includes inputting information for each of the cells into a machine learning model to obtain a parent cell in a row direction of each of the cells and a parent cell in a column direction of each of the cells.

[0009] Optionally, the machine learning model comprises: The method includes: a feature extraction module; a first decision-making module; and a second decision-making module; the feature extraction module is used to perform processing on the information of each of the cells to obtain a characteristic sequence; the first decision-making module is used to obtain a parent cell in a row direction of each of the cells based on the feature sequence; The second decision-making module is used to obtain a column-wise parent cell of each of the cells based on the feature sequence.

[0010] Optionally, the feature extraction module is an encoder of a Transformer model.

[0011] Optionally, the first decision-making module and the second decision-making module are both self-attention modules.

[0012] Optionally, obtaining structural coordinates of each of the cells based on a parent-child relationship in a row direction of each of the cells and a parent-child relationship in a column direction of each of the cells includes: The method includes determining a start row and an end row of each of the cells based on a parent-child relationship in the row direction of each of the cells, and determining a start column and an end column of each of the cells based on a parent-child relationship in the column direction of each of the cells.

[0013] Optionally, at least one tree structure is used to indicate a parent-child relationship in a row direction of each of the cells, and one cell corresponds to one node in the tree, and determining a start row and an end row of each of the cells based on the parent-child relationship in the row direction of each of the cells includes: determining a starting row coordinate b_i corresponding to cell i and determining a number of rows r_i spanned by cell i; determining an ending row coordinate corresponding to cell i as b_i+r_i-1 based on said b_i and said r_i, where If the node corresponding to cell i in the tree in which cell i is located has no sub-branches, then r_i is 1; if there are sub-branches, then the number of rows corresponding to cell i is the sum of the number of rows corresponding to each sub-branch of cell i; If cell i is matched to a table header node, then the start row b_i is equal to the end row of the tree preceding the one in which cell i is located plus one, If the cell i is associated with the kth subnode of node q, then the starting row b_i is equal to b_q+sum1, where b_q is the starting row of node q and sum1 is the sum of the number of rows across all the (k-1) subbranches of node q.

[0014] Optionally, at least one tree structure is used to indicate a parent-child relationship in a row direction of each of the cells, and one cell corresponds to one node in the tree, and determining a start column and an end column of each of the cells based on a parent-child relationship in a column direction of each of the cells includes: determining a number of columns s_i spanned by cell i and determining a starting column coordinate a_i corresponding to cell i; determining an end column coordinate corresponding to cell i as a_i+s_i-1 based on said s_i and a_i, where If the node corresponding to cell i in the tree in which cell i is located has no sub-branches, then s_i is 1; if there are sub-branches, then the number of columns corresponding to cell i is the sum of the number of columns corresponding to each sub-branch of cell i; If the cell i is associated with a table header node, then the start column a_i is equal to the end column of the table header node of the tree preceding the one in which the cell i is located plus one; If the cell i is associated with the jth subnode of node p, then the starting column a_i is equal to a_p+sum2, where a_p is the starting column of node p and sum2 is the sum of the number of columns across all the (j-1) subbranches of node p.

[0015] Optionally, the method further comprises: The method further includes generating a target table based on the structural coordinates of each of the cells and the text within each of the cells.

[0016] According to a second aspect, an embodiment of the present invention provides a table recognition device, the device comprising: an acquisition unit for acquiring a ready image including a table and determining information for each cell in the ready image, the information for each cell including a location of a bounding box for each cell; a first determination unit for obtaining a parent cell in a row direction of each of the cells and a parent cell in a column direction of each of the cells according to information of each of the cells; and a second determination unit for obtaining structural coordinates of each of the cells based on a parent-child relationship of each of the cells in a row direction and a parent-child relationship of each of the cells in a column direction, the structural coordinates including a start row, a start column, an end row, and an end column.

[0017] Optionally, the information for each cell comprises: The data further includes a word embedding vector of the text in each of the cells and / or visual features of each of the cells.

[0018] Optionally, the first determining unit is Information about each of the cells is input into a machine learning model to obtain a parent cell in the row direction of each of the cells and a parent cell in the column direction of each of the cells.

[0019] Optionally, the machine learning model comprises: The method includes: a feature extraction module; a first decision-making module; and a second decision-making module; the feature extraction module is used to perform processing on the information of each of the cells to obtain a characteristic sequence; the first decision-making module is used to obtain a parent cell in a row direction of each of the cells based on the feature sequence; The second decision-making module is used to obtain a column-wise parent cell of each of the cells based on the feature sequence.

[0020] Optionally, the feature extraction module is an encoder of a Transformer model.

[0021] Optionally, the first decision-making module and the second decision-making module are both self-attention modules.

[0022] Optionally, the second determining unit is A start row and an end row of each of the cells are determined based on the parent-child relationship of each of the cells in the row direction, and a start column and an end column of each of the cells are determined based on the parent-child relationship of each of the cells in the column direction.

[0023] Optionally, at least one tree structure is used to indicate a parent-child relationship in a row direction of each of the cells, and one cell corresponds to one node in the tree, and determining a start row and an end row of each of the cells based on the parent-child relationship in the row direction of each of the cells includes: determining a starting row coordinate b_i corresponding to cell i and determining a number of rows r_i spanned by cell i; determining an ending row coordinate corresponding to cell i as b_i+r_i-1 based on said b_i and said r_i, where If the node corresponding to cell i in the tree in which cell i is located has no sub-branches, then r_i is 1; if there are sub-branches, then the number of rows corresponding to cell i is the sum of the number of rows corresponding to each sub-branch of cell i; If cell i is matched to a table header node, then the start row b_i is equal to the end row of the tree preceding the one in which cell i is located plus one, If the cell i is associated with the kth subnode of node q, then the starting row b_i is equal to b_q+sum1, where b_q is the starting row of node q and sum1 is the sum of the number of rows across all the (k-1) subbranches of node q.

[0024] Optionally, at least one tree structure is used to indicate a parent-child relationship in a row direction of each of the cells, and one cell corresponds to one node in the tree, and determining a start column and an end column of each of the cells based on a parent-child relationship in a column direction of each of the cells includes: determining a number of columns s_i spanned by cell i and determining a starting column coordinate a_i corresponding to cell i; determining an end column coordinate corresponding to cell i as a_i+s_i-1 based on said s_i and a_i, where If the node corresponding to cell i in the tree in which cell i is located has no sub-branches, then s_i is 1; if there are sub-branches, then the number of columns corresponding to cell i is the sum of the number of columns corresponding to each sub-branch of cell i; If the cell i is associated with a table header node, then the start column a_i is equal to the end column of the table header node of the tree preceding the one in which the cell i is located plus one; If the cell i is associated with the jth subnode of node p, then the starting column a_i is equal to a_p+sum2, where a_p is the starting column of node p and sum2 is the sum of the number of columns across all the (j-1) subbranches of node p.

[0025] Optionally, the apparatus further comprises: The system further comprises a generating unit for generating a target table based on the structural coordinates of each of the cells and the text within each of the cells.

[0026] According to a third aspect, an embodiment of the present invention provides a device, said device comprising a processor and a memory, The processor executes instructions stored in the memory to cause the device to perform the method of any one of the first aspects described above.

[0027] According to a fourth aspect, an embodiment of the present invention provides a computer readable storage medium comprising instructions for directing a device to perform a method according to any one of the first aspects described above.

[0028] According to a fifth aspect, an embodiment of the present invention provides a computer program product, which when executed on a computer causes the computer to perform a method according to any one of the first aspects described above.

[0029] The embodiment of the present invention has the following advantages over the prior art.

[0030] An embodiment of the present invention provides a table recognition method, in one example, a to-be-processed image including a table is obtained, and information of each cell in the to-be-processed image can be determined, where the information of each cell includes the position of the frame of each cell. Then, based on the information of each cell, a parent cell in the row direction of each cell and a parent cell in the column direction of each cell are obtained. A parent cell in the row direction of each cell is obtained, i.e., a parent-child relationship in the row direction of each cell is obtained. A parent cell in the column direction of each cell is obtained, i.e., a parent-child relationship in the column direction of each cell is obtained. Furthermore, based on the parent-child relationship in the row direction of each cell and the parent-child relationship in the column direction of each cell, a structural coordinate of each cell can be obtained, where the structural coordinate includes a start row, a start column, an end row, and an end column. As can be seen from this, by using the present solution, a structural coordinate of each cell can be obtained based on the parent-child relationship in the row direction of each cell and the parent-child relationship in the column direction of each cell. The parent-child relationships of cells in the row direction and the column direction are simpler than the graph structure, so the present solution can reduce the amount of calculation required to determine the structural coordinates of cells. [Brief description of the drawings]

[0031] In order to more clearly describe the embodiments of the present invention or the technical solutions in the prior art, the following will briefly describe the drawings that need to be used in the embodiments or prior art description, and the drawings in the following description are only some of the embodiments described in the present invention, and it is obvious to those ordinary skilled in the art that they can obtain other drawings according to these drawings without creative efforts. [Figure 1] FIG. 2 is a schematic diagram of a process of a table recognition method provided by an embodiment of the present invention; [Diagram 2] FIG. 2 is a schematic diagram of one exemplary scenario provided by an embodiment of the present invention. [Diagram 3] FIG. 2 is a schematic diagram of another exemplary scenario provided by an embodiment of the present invention. [Figure 4] 1 is a schematic diagram of the structure of a table recognition device provided by an embodiment of the present invention; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0032] In order to allow those skilled in the art to better understand the solution of the present invention, the technical solutions in the embodiments of the present invention are described below clearly and completely in conjunction with the drawings in the embodiments of the present invention, but it is obvious that the described embodiments are only a part of the embodiments of the present invention, and are not all of the embodiments. Based on the embodiments of the present invention, all other embodiments that can be obtained by ordinary skilled in the art without creative efforts belong to the protection scope of the present invention.

[0033] As a result of research, the inventor of the present invention has found that in some methods, cells are regarded as nodes, a table is modeled as a graph structure, and a graph neural network (Graph They discovered that they could use GNNs (Graphical Neural Networks) to model the relationships between cells to determine, for example, whether two cells are in the same row and the same column, and then use logical processing methods to obtain the structural coordinates of the table based on whether the cells are in the same row and whether the cells are in the same column.

[0034] In the case of a graph structure, any relationship may exist between any two nodes, and therefore it is not difficult to understand that the relationships between nodes in the graph structure obtained by modeling with a graph are more complicated, and even a ring structure may exist. Therefore, in order to obtain the structural coordinates of the table using the graph structure, it is necessary to use a more complicated logical processing method, which requires consuming more computational resources.

[0035] To solve the above problem, the embodiment of the present invention provides a table recognition method, which can obtain the structural coordinates of each cell based on the parent-child relationship of each cell in the row direction and the parent-child relationship of each cell in the column direction. The parent-child relationship of each cell in the row direction and the parent-child relationship of each cell in the column direction are simpler than the graph structure. Therefore, the present solution can use a simpler logical processing method to determine the structural coordinates of each cell, and reduce the consumption of calculation resources.

[0036] Various non-limiting embodiments of the present invention will now be described in detail in conjunction with the drawings.

[0037] Exemplary Methods Referring to Fig. 1, this figure is a schematic diagram of a process of a table recognition method provided by an embodiment of the present invention. The method shown in Fig. 1 may be executed by a client or a server, and the embodiment of the present invention is not specifically limited. In this embodiment, the method may include, for example, the following steps S101 to S103.

[0038] In S101, a ready image including a table is obtained, and information of each cell in the ready image is determined, where the information of each cell includes a bounding box position of each cell.

[0039] In the embodiments of the present invention, when specifically realizing the acquisition of the image awaiting processing, the image awaiting processing may be acquired by capturing an image using a photographing device, the image awaiting processing may be acquired from a network resource, or the image awaiting processing may be acquired in other ways, and the embodiments of the present invention are not specifically limited.

[0040] In an embodiment of the present invention, the information of each cell refers to information related to each cell.

[0041] In one example, Optical Character Recognition In the method, the text in the image to be processed is recognized by using Optical Character Recognition (OCR) technology, and then the position of the frame of each cell is determined based on the recognized text. Here, for convenience of explanation, any one of the cells is referred to as a "first cell." Then, the position of the frame of the first cell is the position of a text box that surrounds the text in the first cell.

[0042] In one example, the information of each cell may include the position of the box of each cell as well as the word embedding vector of the text in each cell. As one example, the text in each unit may be recognized and obtained using OCR technology, and then a word embedding vector model such as bert or word2vec may be used to process the text in each cell to obtain the word embedding vector of the text in each cell.

[0043] In another example, the information for each cell may include visual characteristics of each cell in addition to the location of the bounding box for each cell, where the visual characteristics of each cell may be characteristics obtained by performing a calculation (e.g., a convolution calculation) on the image region in which each cell is located, and the visual characteristics of each cell may reflect visual information such as a background color and / or texture of each cell.

[0044] In S102, based on the information of each of the cells, a parent cell in the row direction of each of the cells and a parent cell in the column direction of each of the cells are obtained.

[0045] In one example, a parent cell in the row direction of each of the cells and a parent cell in the column direction of each of the cells can be determined based on the size relationship between the horizontal axis coordinates of each of the cells and the size relationship between the vertical axis coordinates of each of the cells.

[0046] In another example, a machine learning model can be pre-trained and utilized to determine each cell's row parent cell and each cell's column parent cell. Specifically, information for each cell can be input into the machine learning model to obtain each cell's row parent cell and each cell's column parent cell.

[0047] Regarding the machine learning model, in one example, it should be explained that the machine learning model may include a feature extraction module, a first decision-making module, and a second decision-making module, and the feature extraction module is used to perform processing on the input information of each cell to obtain a feature sequence, where the feature sequence is used as an input to both the first decision-making module and the second decision-making module. The first decision-making module is used to obtain a parent cell in a row direction of each cell based on the feature sequence, and the second decision-making module is used to obtain a parent cell in a column direction of each cell based on the feature sequence.

[0048] In one example, the machine learning model may be used for natural language processing, considering that the output of the machine learning model depends on the input of the machine learning model in that the machine learning model is used to determine the parent cell in the row direction of each node and the parent cell in the column direction of each node. The computation method of PointerNet, which is commonly used in neural network processing (NLP), can be used. In addition, the conventional PointerNet is based on a recurrent neural network (RNN). In order to enhance the long-distance dependency modeling ability of the machine learning model, the machine learning model can accurately determine the parent cell in the row direction of each cell and the parent cell in the column direction of each cell even when the number of cells in the table is larger, and in an embodiment of the present invention, the PointerNet can be optimized into an encoder based on Transformer. In other words, in one embodiment, the feature extraction module is an encoder of a Transformer model.

[0049] Here, the encoder may include a self-attention module and a feedforward neural network module, the calculation methods of which are similar to those in the encoder of the conventional Transformer model, and will not be described in detail here.

[0050] In one example, after the information of each cell passes through the feature extraction module, feature matrices Q and K can be obtained, where assuming that the information input to the machine learning model contains information of n cells, Q and K are both n*m matrices.

[0051] In one example, the first decision-making module and the second decision-making module are both self-attention modules.

[0052] In one example, the first decision module calculates a relation matrix QK for Q and K. T Then, we can perform a softmax operation on each row of the associated matrix to obtain an n*n α matrix, where the element αij is used to denote the probability that cell j is a parent cell of cell i in the row direction.

[0053] Similarly, the second decision-making module can also calculate an n*n β matrix based on Q and K, and the element β of the β matrix ij is used to denote the probability that cell j is a parent cell of cell i in the column direction.

[0054] It is necessary to explain that for a cell as a root node, one virtual node (eg, labelled -1) can be introduced to represent the parent node of the root node.

[0055] The parent nodes in the row direction of each cell are described in conjunction with Figure 2, which is a schematic diagram of one example scenario provided by an embodiment of the present invention, where cells are represented by nodes.

[0056] As shown in Figure 2, the parent node of node 0 is node-1, the parent nodes of node 1 and node 2 are both node 0, the parent node of node 3 is node 2, and the parent node of node 4 is node 3. Based on the parent nodes in the row direction of each node shown in Figure 2, the tree structure shown in Figure 2 can be obtained.

[0057] Regarding the parent node in the column direction of each cell, and determining the tree structure based on the parent node in the column direction of each cell, no further examples will be given here.

[0058] In one example, the machine learning model can be trained using a training table whose structural coordinates are known. Since the structural coordinates of the training table are known, the cell information of each cell in the training table and the parent cell in the row direction of each cell are known, and the parent cell in the column direction of each cell in the training table is also known. Therefore, the machine learning model mentioned above can be trained based on the training table.

[0059] In S103, based on the parent-child relationship of each of the cells in the row direction and the parent-child relationship of each of the cells in the column direction, structural coordinates of each of the cells are obtained, where the structural coordinates include a start row, a start column, an end row, and an end column.

[0060] After executing S102, the parent cell in the row direction of each cell is obtained, that is, the parent-child relationship in the row direction of each cell is obtained, and accordingly, the parent cell in the column direction of each cell is obtained, that is, the parent-child relationship in the column direction of each cell is obtained.Furthermore, by executing S103, the structural coordinates of each cell can be obtained.

[0061] Here, the parent-child relationship of each cell in the row direction may be reflected as a row forest, which may include a plurality of trees, and the parent-child relationship of each cell in the column direction may be reflected as a column forest, which may include a plurality of trees.

[0062] In one example, when S103 is specifically implemented, the start row and end row of each of the cells can be determined based on the parent-child relationship of each of the cells in the row direction, and the start column and end column of each of the cells can be determined based on the parent-child relationship of each of the cells in the column direction.

[0063] First, a specific embodiment of "determining the start row and end row of each of the cells based on the parent-child relationship in the row direction of each of the cells" will be described.

[0064] For a cell i of the respective cells, a starting row coordinate b_i corresponding to cell i can be determined, a number of rows r_i spanned by cell i can be determined, and an ending row coordinate of cell i can be determined as b_i+r_i-1 based on b_i and r_i, where cell i is any one of the respective cells.

[0065] For r_i, we need to explain: If the node corresponding to cell i in the tree in which cell i is located has no sub-branches, it indicates that cell i is the finest-grained cell. In this case, r_i is 1, If the node corresponding to cell i in the tree in which cell i is located has sub-branches, i.e., cell i contains multiple sub-cells, then the number of rows corresponding to cell i is the sum of the number of rows corresponding to each of the sub-branches of cell i.

[0066] In determining the number of rows r_i spanned by cell i, a right-to-left approach may be used to determine the number of rows spanned by each node in the tree in which said cell i is located.

[0067] For b_i, we need to explain When cell i is associated with a table header node, the start row b_i is equal to the end row of the table header node of the tree preceding the one in which cell i is located plus one, where the parent node of the table header node is the virtual node.

[0068] As described above, a row forest can be obtained based on the parent-child relationship in the row direction of each cell, and the row forest can include multiple trees. In one example, the multiple tree structures can be sorted in order from smallest to largest ordinates of table header nodes in the multiple tree structures. The one tree before the tree in which the cell i is located refers to the one tree sorted before the tree in which the cell i is located after sorting the multiple tree structures in the above-mentioned order.

[0069] If the cell i is associated with the kth subnode of node q, then the starting row b_i is equal to b_q+sum1, where b_q is the starting row of node q and sum1 is the sum of the number of rows across all the (k-1) subbranches of node q, where Node q may contain one subnode, where k is 1, sum1=0, b_i=b_q, Node q may also include multiple subnodes, in which case the value of k may be greater than or equal to 1. When the value of k is 1, sum1=0 and b_i=b_q, and when the value of k is greater than 1, b_i=b_q+sum1.

[0070] The starting row of the node corresponding to each cell can be determined in order from left to right, i.e., the starting row of the table header node can be determined first, and then the starting row of each subnode of the table header node can be determined, and the starting row of node q can be obtained accordingly, and the starting row of cell i, which is the kth subnode corresponding to node q, can also be determined accordingly.

[0071] The "determining the start row and the end row of each of the cells based on the parent-child relationship in the row direction of each of the cells" will be described in conjunction with Fig. 3. Fig. 3 is a schematic diagram of another exemplary scenario provided by an embodiment of the present invention.

[0072] As shown in Fig. 3, for an image containing a table as shown in Fig. 3, a machine learning model can be used to obtain a row forest in the lower left area of ​​Fig. 3. The row forest includes four trees. One node in Fig. 3 corresponds to one cell.

[0073] Using a right-to-left approach, determine the number of rows spanned by each node in the first tree, where: Since node 5 has no branches in the first tree, the number of rows spanned by node 5 is 1, Similarly, since nodes 6, 7, and 8 have no branches in the first tree, the number of rows that span nodes 6, 7, and 8 is 1. Node 1 has two branches in the first tree, one with node 5 as the root node and the other with node 6 as the root node. The number of lines spanned by each of these two branches is 1, so the number of lines spanned by node 1 is 2.

[0074] Since node 1 is the table header node of the first tree, the start row of node 1 is 0 (or 1; in this invention, an example is described in which the start row of node 1 is 0), and the end row of node 1 is 0+2-1=1.

[0075] Since node 5 is the first subnode of node 1, the start row of node 5 is equal to the start row of node 1, i.e., the start row of node 5 is 0 and the number of rows spanned by node 5 is 1, therefore, the end row of node 5 is 0.

[0076] Since node 6 is the 2nd subnode of node 1, the startrow of node 6 is equal to the number of lines spanned by the startrow of node 1 to the 1st subnode of node 1, i.e., the startrow of node 6 is equal to 0+1=1, and since the number of lines spanned by node 6 is 1, the endrow of node 6 is 1.

[0077] Since node 7 is the first subnode of node 6, the start row of node 7 is equal to the start row of node 6, i.e., the start row of node 7 is 1 and the number of rows spanned by node 7 is 1, therefore, the end row of node 7 is 1.

[0078] Since node 8 is the first subnode of node 7, the start row of node 8 is equal to the start row of node 7, i.e., the start row of node 8 is 1 and the number of rows spanned by node 8 is 1, therefore, the end row of node 8 is 1.

[0079] Using a right-to-left approach, determine the number of rows spanned by each node in the second tree, where: Since nodes 2, 9, 10, and 11 have no branches in the second tree, the number of rows that span nodes 2, 9, 10, and 11 is 1. Since node 2 is the table header node of the second tree, the start row of node 2 is 2 (adding 1 to the end row of the table header node of the first tree, which is 1), and the end row of node 2 is 2+1-1=2.

[0080] Since node 9 is the first subnode of node 2, the start row of node 9 is equal to the start row of node 2, i.e., the start row of node 9 is 2 and the number of rows spanned by node 9 is 1, therefore, the end row of node 9 is 2.

[0081] Since node 10 is the first subnode of node 9, the start row of node 10 is equal to the start row of node 9, i.e., the start row of node 10 is 2 and the number of rows spanned by node 10 is 1, therefore, the end row of node 10 is 2.

[0082] Since node 11 is the first subnode of node 10, the start row of node 11 is equal to the start row of node 10, i.e., the start row of node 11 is 2 and the number of rows spanned by node 11 is 1, therefore, the end row of node 11 is 2.

[0083] Accordingly, it can be obtained that the starting row is 3 and the ending row is 3 for node 3, node 12, node 13, and node 14, and the starting row is 4 and the ending row is 4 for node 4, node 15, node 16, and node 17.

[0084] Next, a specific embodiment of "determining the start column and the end column of each of the cells based on the parent-child relationship in the column direction of each of the cells" will be described.

[0085] For each of the cells, for cell i, a number of columns s_i that span cell i can be determined, and a start column coordinate a_i corresponding to cell i can be determined, and then an end column coordinate corresponding to cell i is determined as a_i+s_i-1 based on s_i and a_i.

[0086] Regarding the above s_i, what needs to be explained is that If the node corresponding to cell i in the tree in which cell i is located has no sub-branches, it indicates that cell i is the finest-grained cell, and in this case, s_i is 1; If the node corresponding to cell i in the tree in which cell i is located has sub-branches, then the number of columns corresponding to cell i is the sum of the corresponding numbers of columns of each of the sub-branches of cell i.

[0087] What needs to be explained about a_i is that When cell i is associated with a table header node, the start column a_i is equal to the end column of the table header node of the tree preceding the one in which cell i is located plus one.

[0088] As described above, a column forest can be obtained based on the parent-child relationship in the column direction of each cell, and the column forest can include multiple trees. In one example, the multiple tree structures can be sorted in order from smallest to largest abscissas of table header nodes in the multiple tree structures. The one tree before the tree in which cell i is located is the one tree sorted before the tree in which cell i is located after sorting the multiple tree structures in the above-mentioned order.

[0089] If the cell i is associated with the j-th subnode of node p, then the starting column a_i is equal to a_p+sum2, where a_p is the starting column of node p and sum2 is the sum of the number of columns across all the (j-1) sub-branches of node p, where Node p can contain one subnode, then j has a value of 1, sum2=0, a_i=a_p, A node p may also contain multiple subnodes, in which case the value of j may be greater than or equal to 1. When the value of j is equal to 1, sum2=0 and a_i=a_p, and when the value of j is greater than 1, a_i=a_p+sum2.

[0090] The starting column of the node corresponding to each cell can be determined in order from top to bottom, i.e., the starting column of the table header node can be determined first, and then the starting column of each subnode of the table header node can be determined, and the starting column of node p can be obtained accordingly, and the starting column of cell i, which is the jth subnode corresponding to node p, can also be determined accordingly.

[0091] The "determining the start column and the end column of each of the cells based on the parent-child relationship in the column direction of each of the cells" will be described with an example in conjunction with FIG.

[0092] As shown in Figure 3, for an image containing a table as shown in Figure 3, a machine learning model can be used to obtain a row forest in the lower right region of Figure 3. The row forest includes two trees.

[0093] Use a bottom-up approach to determine the number of columns spanned by each node in the first tree, where: Since nodes 4, 3, 2, and 1 have no branches in the first tree, the number of columns that spans nodes 4, 3, 2, and 1 is 1. Since node 1 is the table header node of the first tree, the start column of node 1 is 0 (or 1, and in this invention we will explain an example where the start column of node 1 is 0), and the end column of node 1 is 0+1-1=0.

[0094] Since node 2 is the first subnode of node 1, the start column of node 2 is equal to the start column of node 1, i.e., the start column of node 2 is 0 and the number of columns spanned by node 2 is 1, therefore, the end column of node 2 is 0.

[0095] Since node 3 is the first subnode of node 2, the start column of node 3 is equal to the start column of node 2, i.e., the start column of node 3 is 0 and the number of columns spanned by node 3 is 1, therefore, the end column of node 3 is 0.

[0096] Since node 4 is the first subnode of node 3, the start column of node 4 is equal to the start column of node 3, i.e., the start column of node 4 is 0 and the number of columns spanned by node 4 is 1, therefore, the end column of node 4 is 0.

[0097] Use a bottom-up approach to determine the number of columns spanned by each node in the second tree, where: Since node 6, node 7, node 8, node 9, node 10, node 11, node 12, node 13, node 14, node 15, node 16, and node 17 do not have any sub-branches, the number of columns spanning node 6, node 7, node 8, node 9, node 10, node 11, node 12, node 13, node 14, node 15, node 16, and node 17 is all 1, The number of columns spanned by node 5 is the sum of the numbers of columns spanned by its three subnodes (node ​​6, node 7, and node 8), i.e., the number of columns spanned by node 5 is three.

[0098] In top-down order, determine the start and end columns of each node in the second tree.

[0099] The start column of node 5 is the end column of the table header node of the first tree plus 1, i.e., we add 1 to the end column of node 1, which is 0, to get the start column of node 5 to be 1, and since the number of columns spanned by node 5 is 3, the end column of node 5 is 1+3-1=3.

[0100] Since node 6 corresponds to the first subnode of node 5, the start column of node 6 is equal to the start column of node 5, i.e., the start column of node 6 is 1 and the number of columns spanned by node 6 is 1, therefore, the end column of node 6 is also 1.

[0101] Since node 9 is the first subnode of node 6, the start column of node 9 is equal to the start column of node 6, i.e., the start column of node 9 is 1 and the number of columns spanned by node 9 is 1, therefore, the end column of node 9 is also 1. Accordingly, we can obtain that the start column and end column of node 12 and node 15 are both 1.

[0102] Since node 7 corresponds to the second subnode of node 5, the start column of node 7 is equal to the start column of node 5, which is 1, plus the number of columns spanned in the first subbranch of node 5, which is 1, so the start column of node 7 is 2 and the number of columns spanned by node 7 is 1, so the end column of node 7 is also 2.

[0103] Since node 10 is the first subnode of node 7, the start column of node 10 is equal to the start column of node 7, i.e., the start column of node 10 is 2 and the number of columns spanned by node 10 is 1, therefore, the end column of node 10 is also 2. Accordingly, we can obtain that the start column and end column of node 13 and node 16 are both 2.

[0104] Since node 8 corresponds to the third subnode of node 5, the start column of node 8 is equal to the start column of node 5, 1, plus the sum of the number of columns spanned by the two branches before node 5, 2; i.e., the start column of node 8 is 3 and the number of columns spanned by node 8 is 1; therefore, the end column of node 8 is also 3.

[0105] Since node 11 is the first subnode of node 8, the start column of node 11 is equal to the start column of node 8, i.e., the start column of node 11 is 3 and the number of columns spanned by node 11 is 1, therefore, the end column of node 11 is also 3. Accordingly, we can obtain that the start column and end column of node 14 and node 17 are both 3.

[0106] As can be seen from the above description, by using this solution, the structural coordinates of each cell can be obtained based on the parent-child relationship of each cell in the row direction and the parent-child relationship of each cell in the column direction. The parent-child relationship of cells in the row direction and the parent-child relationship of cells in the column direction are simpler than the graph structure. Therefore, by using this solution, the amount of calculation for determining the structural coordinates of cells can be reduced.

[0107] In one example, after determining the structure coordinates of each cell, the coordinate structure and the text in each cell can be combined to generate a target table, where the text in each cell can be obtained using OCR techniques.

[0108] The embodiment of the present invention does not specifically limit the format of the target table, and the target table may be in Excel format, Word format, or other formats, and is not limited thereto.

[0109] If the target table is in excel format, it is also possible to determine the total number of columns and rows of the table according to the structural coordinates of each cell, so that when generating the target table, firstly divide the cells according to the total number of columns and the total number of rows, and then obtain each cell in the target table by combining the operations of the cells according to the structural coordinates of each cell, where: The total number of columns is the maximum value of the end column of each cell minus the minimum value of the start column of each cell plus 1. For the scenario shown in Figure 3, the maximum value of the end column of each cell is 3 and the minimum value of the start column of each cell is 0, so the total number of columns is 3-0+1=4.

[0110] The total number of rows is the maximum end row of each cell minus the minimum start row of each cell plus 1. For the scenario shown in Figure 3, the maximum end row of each cell is 4 and the minimum start row of each cell is 0, so the total number of rows is 4-0+1=5.

[0111] Exemplary Devices Based on the method provided by the above embodiment, the embodiment of the present invention further provides an apparatus, which is described below in conjunction with the drawings.

[0112] 4, which is a schematic diagram of the structure of a table recognition device provided by an embodiment of the present invention. The device 400 may, for example, specifically include an acquisition unit 401, a first determination unit 402 and a second determination unit 403.

[0113] The acquisition unit 401 acquires a raw image including a table, and determines information of each cell in the raw image, the information of each cell including a position of a bounding box of each cell; The first determination unit 402 obtains a parent cell in a row direction of each of the cells and a parent cell in a column direction of each of the cells according to the information of each of the cells; The second determination unit 403 obtains structural coordinates of each of the cells based on the parent-child relationship of each of the cells in the row direction and the parent-child relationship of each of the cells in the column direction, where the structural coordinates include a start row, a start column, an end row, and an end column.

[0114] Optionally, the information for each cell comprises: The data further includes a word embedding vector of the text in each of the cells and / or visual features of each of the cells.

[0115] Optionally, the first determining unit 402 is Information about each of the cells is input into a machine learning model to obtain a parent cell in the row direction of each of the cells and a parent cell in the column direction of each of the cells.

[0116] Optionally, the machine learning model comprises: The method includes: a feature extraction module; a first decision-making module; and a second decision-making module; the feature extraction module is used to perform processing on the information of each of the cells to obtain a characteristic sequence; the first decision-making module is used to obtain a parent cell in a row direction of each of the cells based on the feature sequence; The second decision-making module is used to obtain a column-wise parent cell of each of the cells based on the feature sequence.

[0117] Optionally, the feature extraction module is an encoder of a Transformer model.

[0118] Optionally, the first decision-making module and the second decision-making module are both self-attention modules.

[0119] Optionally, the second determining unit 403 is A start row and an end row of each of the cells are determined based on the parent-child relationship of each of the cells in the row direction, and a start column and an end column of each of the cells are determined based on the parent-child relationship of each of the cells in the column direction.

[0120] Optionally, at least one tree structure is used to indicate a parent-child relationship in a row direction of each of the cells, and one cell corresponds to one node in the tree, and determining a start row and an end row of each of the cells based on the parent-child relationship in the row direction of each of the cells includes: determining a starting row coordinate b_i corresponding to cell i and determining a number of rows r_i spanned by cell i; determining an ending row coordinate corresponding to cell i as b_i+r_i-1 based on said b_i and said r_i, where If the node corresponding to cell i in the tree in which cell i is located has no sub-branches, then r_i is 1; if there are sub-branches, then the number of rows corresponding to cell i is the sum of the number of rows corresponding to each sub-branch of cell i; If cell i is matched to a table header node, then the start row b_i is equal to the end row of the tree preceding the one in which cell i is located plus one, If the cell i is associated with the kth subnode of node q, then the starting row b_i is equal to b_q+sum1, where b_q is the starting row of node q and sum1 is the sum of the number of rows across all the (k-1) subbranches of node q.

[0121] Optionally, at least one tree structure is used to indicate a parent-child relationship in a row direction of each of the cells, and one cell corresponds to one node in the tree, and determining a start column and an end column of each of the cells based on a parent-child relationship in a column direction of each of the cells includes: determining a number of columns s_i spanned by cell i and determining a starting column coordinate a_i corresponding to cell i; determining an end column coordinate corresponding to cell i as a_i+s_i-1 based on said s_i and a_i, where If the node corresponding to cell i in the tree in which cell i is located has no sub-branches, then s_i is 1; if there are sub-branches, then the number of columns corresponding to cell i is the sum of the number of columns corresponding to each sub-branch of cell i; If the cell i is associated with a table header node, then the start column a_i is equal to the end column of the table header node of the tree preceding the one in which the cell i is located plus one; If the cell i is associated with the jth subnode of node p, then the starting column a_i is equal to a_p+sum2, where a_p is the starting column of node p and sum2 is the sum of the number of columns across all the (j-1) subbranches of node p.

[0122] Optionally, the apparatus further comprises: The system further comprises a generating unit for generating a target table based on the structural coordinates of each of the cells and the text within each of the cells.

[0123] The device 400 is an device corresponding to the table recognition method provided by the above-mentioned method embodiments, and the specific implementation of each unit of the device 400 is the same idea as the table recognition method described in the above-mentioned method embodiments. Therefore, for the specific implementation of each unit of the device 400, please refer to the relevant description parts of the above-mentioned method embodiments, and will not be repeated here.

[0124] An embodiment of the present invention further provides a device, the device comprising a processor and a memory; The processor executes instructions stored in the memory to cause the device to perform the table recognition method provided by the method embodiment described above.

[0125] An embodiment of the present invention provides a computer-readable storage medium, comprising instructions for directing a device to perform a table recognition method as provided by the method embodiment described above.

[0126] An embodiment of the present invention further provides a computer program product, which, when executed on a computer, causes the computer to perform the table recognition method provided by the method embodiment described above.

[0127] Those skilled in the art will readily conceive other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. The present invention covers any modification, use or adaptation of the present invention, which modification, use or adaptation follows the general principles of the present invention and includes common knowledge or customary means known in the art that are not disclosed in this disclosure. It is intended that the specification and examples be considered as exemplary only, with the true scope and spirit of the invention being indicated by the following claims.

[0128] It should be understood that the present invention is not limited to the exact construction described above and illustrated in the drawings, but various modifications and changes can be made without departing from the scope of the present invention, which is limited only by the appended claims.

[0129] The above is only a preferred embodiment of the present invention, and is not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. obtaining an image to be processed that includes a table and determining information for each cell in the image to be processed, the information for each cell including a location of a bounding box for each cell; obtaining a parent cell in a row direction of each of the cells and a parent cell in a column direction of each of the cells based on information of each of the cells; obtaining structural coordinates of each of the cells based on a parent-child relationship in a row direction of each of the cells and a parent-child relationship in a column direction of each of the cells, the structural coordinates including a start row, a start column, an end row, and an end column; A table recognition method comprising:

2. The information for each cell is:

2. The method of claim 1, further comprising: a word embedding vector of the text in each of the cells; and / or visual features of each of the cells.

3. Obtaining a parent cell in a row direction of each of the cells and a parent cell in a column direction of each of the cells based on the information of each of the cells, 3. The method of claim 1 or 2, comprising inputting information for each cell into a machine learning model to obtain a row-wise parent cell for each cell and a column-wise parent cell for each cell.

4. The machine learning model is A feature extraction module, a first decision-making module, and a second decision-making module, the feature extraction module is used to perform processing on the information of each of the cells to obtain a characteristic sequence; the first decision-making module is used to obtain a parent cell in a row direction of each of the cells based on the feature sequence; 4. The method of claim 3, wherein the second decision-making module is used to obtain a column-wise parent cell of each of the cells based on the feature sequence.

5. The method of claim 4 , wherein the feature extraction module is an encoder of a Transformer model.

6. 5. The method of claim 4, wherein the first decision-making module and the second decision-making module are both self-attention modules.

7. obtaining structural coordinates of each of the cells based on a parent-child relationship in a row direction of the each of the cells and a parent-child relationship in a column direction of the each of the cells, 2. The method of claim 1, further comprising: determining a start row and an end row of each of the cells based on a parent-child relationship in a row direction of each of the cells; and determining a start column and an end column of each of the cells based on a parent-child relationship in a column direction of each of the cells.

8. At least one tree structure is used to indicate a parent-child relationship in a row direction of each of the cells, with one cell corresponding to one node in the tree, and determining a start row and an end row of each of the cells based on the parent-child relationship in the row direction of each of the cells includes: determining a starting row coordinate b_i corresponding to cell i and determining a number of rows r_i spanned by cell i; determining an end row coordinate corresponding to cell i as b_i+r_i-1 based on said b_i and said r_i; where r_i is 1 if the node corresponding to cell i in the tree in which cell i is located has no sub-branches, and if there are sub-branches, the number of rows corresponding to cell i is the sum of the number of rows corresponding to each of the sub-branches of cell i; If cell i corresponds to a table header node, then the start row b_i is equal to the end row of the tree preceding the one in which cell i is located plus one; 8. The method of claim 7, wherein if the cell i corresponds to the kth subnode of node q, then the starting row b_i is equal to b_q+sum1, where b_q is the starting row of node q and sum1 is the sum of the total number of rows spanned by the (k-1) sub-branches of node q.

9. At least one tree structure is used to indicate a row-wise parent-child relationship of each of the cells, with one cell corresponding to one node in the tree, and determining a start column and an end column of each of the cells based on the column-wise parent-child relationship of each of the cells includes: determining a number of columns s_i spanned by cell i and determining a starting column coordinate a_i corresponding to cell i; determining an end column coordinate corresponding to cell i as a_i+s_i−1 based on said s_i and a_i; where s_i is 1 if the node corresponding to cell i in the tree in which cell i is located has no sub-branches, and if there are sub-branches, the number of columns corresponding to cell i is the sum of the number of columns corresponding to each of the sub-branches of cell i; If the cell i corresponds to a table header node, then the start column a_i is equal to the end column of the table header node of the tree preceding the one in which the cell i is located plus one; 8. The method of claim 7, wherein when the cell i corresponds to the j-th subnode of node p, the starting column a_i is equal to a_p+sum2, where a_p is the starting column of node p, and sum2 is the sum of the total spanned column numbers of the (j-1) sub-branches of node p.

10. generating a target table based on the structural coordinates of each of the cells and the text within each of the cells.

2. The method of claim 1 .

11. an acquisition unit for acquiring an image to be processed comprising a table and for determining information of each cell in the image to be processed, the information of each cell including a position of a perimeter of each cell; a first determination unit for obtaining a row-direction parent cell of each of the cells and a column-direction parent cell of each of the cells based on information of each of the cells; a second determination unit for obtaining structural coordinates of each of the cells based on a parent-child relationship in a row direction of each of the cells and a parent-child relationship in a column direction of each of the cells, the structural coordinates including a start row, a start column, an end row, and an end column; A table recognition device comprising:

12. A device, comprising: A processor and a memory, A device, characterized in that the processor is adapted to cause the device to carry out the method according to any one of claims 1 to 10 by executing instructions stored in the memory.

13. A computer readable storage medium comprising instructions for directing a device to carry out the method according to any one of claims 1 to 10.

14. A computer program product which, when executed on a computer, causes the computer to carry out the method according to any one of claims 1 to 10.

Images