Table recognition method, table recognition model training method, device and equipment

By performing preset size expansion and attribute extraction on table images, combined with feature extraction and decoder technology, the accuracy and universality of table recognition in the prior art are solved, and efficient identification and reconstruction of different table scenarios are achieved.

CN116645685BActive Publication Date: 2024-11-26BEIJING BAIDU NETCOM SCI & TECH CO LTD
View PDF 9 Cites 0 Cited by

Patent Information

Application Number
CN202310369035.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-04-07
Publication Date
2024-11-26
Estimated Expiration
2043-04-07

AI Technical Summary

Technical Problem

The prior art is difficult to effectively identify tables with tilted, distorted or merged cells in table structure recognition, and lacks universality and cannot adapt to different table scenarios.

Method used

By augmenting the preset number of rows and total columns to identify the table image, the table attribute information includes row validity, column validity, table vertex coordinates and adjacent vertex connectivity, and table reconstruction is performed based on these attributes. Feature extractors and decoders improve recognition accuracy, introducing row and column enhancement decoders to enhance the performance of the recognition model.

Benefits of technology

It improves the accuracy and versatility of table recognition, can be applied to table scenarios of different sizes and structures, simplifies the table data extraction process, and improves office efficiency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN116645685B_ABST
    Figure CN116645685B_ABST
Patent Text Reader

Abstract

The present disclosure provides a table recognition method, a training method, a device and equipment for a table recognition model, which relate to the field of artificial intelligence technology, specifically computer vision, OCR, deep learning and other technical fields, and can be applied to scenarios such as smart finance and smart government affairs. The method includes: expanding the table size of the table image to be recognized according to the preset total number of rows and total number of columns to obtain the expanded target table image; the total number of rows and the total number of columns are both fixed values; performing table recognition on the target table image to obtain the table attribute information of the target table image; the table attribute information includes row validity information, column validity information, coordinates of table vertices and connectivity between adjacent table vertices; reconstructing the table according to the table attribute information of the target table image to obtain the cell information in the table image to be recognized. The above technical solution can improve the accuracy of table recognition.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, specifically computer vision, OCR, deep learning and other technical fields, and can be applied to scenarios such as smart finance and smart government affairs. Specifically, it relates to a table recognition method, a table recognition model training method, a device and equipment. Background Art

[0002] With the rapid development of artificial intelligence (AI), AI has been widely used in computer vision technology, speech recognition technology, natural language processing technology, deep learning, big data processing technology and other fields.

[0003] Table structure recognition technology based on artificial intelligence has also been widely used in different scenarios. Table structure recognition can automatically transcribe and analyze data such as financial statements and billing tables in images, which is of great significance for efficient office work and reducing inefficient labor. Therefore, how to recognize the table structure is very important. Summary of the invention

[0004] The present disclosure provides a table recognition method, a table recognition model training method, a device and equipment.

[0005] According to one aspect of the present disclosure, a table recognition method is provided, comprising:

[0006] Expanding the table size of the table image to be recognized according to a preset total number of rows and a total number of columns to obtain an expanded target table image; the total number of rows and the total number of columns are both fixed values;

[0007] Performing table recognition on the target table image to obtain table attribute information of the target table image; the table attribute information includes row validity information, column validity information, coordinates of table vertices and connectivity between adjacent table vertices;

[0008] The table is reconstructed according to the table attribute information of the target table image to obtain the cell information in the table image to be identified.

[0009] According to another aspect of the present disclosure, a method for training a table recognition model is provided, comprising:

[0010] Expanding the table size of the sample table image according to a preset total number of rows and a total number of columns to obtain an expanded auxiliary table image; the total number of rows and the total number of columns are both fixed values;

[0011] Using the table recognition model to be trained to perform table recognition on the auxiliary table image to obtain table attribute information of the auxiliary table image; the table attribute information of the auxiliary table image includes validity information of predicted rows, validity information of predicted columns, coordinates of predicted table vertices, and connectivity between adjacent predicted table vertices in the auxiliary table image;

[0012] The table recognition model is trained according to the table attribute information marked by the sample table image and the table attribute information of the auxiliary table image; the table attribute information marked by the sample table image includes the validity information of the marked rows in the sample table image, the validity information of the marked columns, the coordinates of the marked table vertices and the connectivity between adjacent marked table vertices.

[0013] According to one aspect of the present disclosure, there is provided a table recognition device, comprising:

[0014] A target table expansion module is used to expand the table size of the table image to be recognized according to a preset total number of rows and a total number of columns to obtain an expanded target table image; the total number of rows and the total number of columns are both fixed values;

[0015] A target table recognition module, used to perform table recognition on the target table image to obtain table attribute information of the target table image; the table attribute information includes row validity information, column validity information, coordinates of table vertices and connectivity between adjacent table vertices;

[0016] The target table reconstruction module is used to reconstruct the table according to the table attribute information of the target table image to obtain the cell information in the table image to be identified.

[0017] According to one aspect of the present disclosure, a training device for a table recognition model is provided, comprising:

[0018] The auxiliary table expansion module expands the table size of the sample table image according to the preset total number of rows and total number of columns to obtain an expanded auxiliary table image; the total number of rows and total number of columns are both fixed values;

[0019] an auxiliary table recognition module, which uses a table recognition model to be trained to perform table recognition on the auxiliary table image to obtain table attribute information of the auxiliary table image; the table attribute information of the auxiliary table image includes validity information of predicted rows, validity information of predicted columns, coordinates of predicted table vertices, and connectivity between adjacent predicted table vertices in the auxiliary table image;

[0020] The table model training module is used to train the table recognition model according to the table attribute information marked by the sample table image and the table attribute information of the auxiliary table image; the table attribute information marked by the sample table image includes the validity information of the marked rows in the sample table image, the validity information of the marked columns, the coordinates of the marked table vertices and the connectivity between adjacent marked table vertices.

[0021] According to another aspect of the present disclosure, there is provided an electronic device, the electronic device comprising:

[0022] at least one processor; and

[0023] a memory communicatively connected to the at least one processor; wherein,

[0024] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method provided by any embodiment of the present disclosure.

[0025] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to execute the method provided by any embodiment of the present disclosure.

[0026] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0028] Figure 1a is a flow chart of a table recognition method provided according to an embodiment of the present disclosure;

[0029] Figure 1b is a table schematic diagram of a table image to be recognized provided according to an embodiment of the present disclosure;

[0030] Figure 1c is a table schematic diagram of a target table image provided according to an embodiment of the present disclosure;

[0031] Figure 2a is a flow chart of another table recognition method provided according to an embodiment of the present disclosure;

[0032] Figure 2b is a structural diagram of a table recognition model provided according to an embodiment of the present disclosure;

[0033] Figure 2cis a structural schematic diagram of another table recognition model provided according to an embodiment of the present disclosure;

[0034] Figure 3 is a flowchart of another table recognition method provided according to an embodiment of the present disclosure;

[0035] Figure 4a is a flowchart of another table recognition method provided according to an embodiment of the present disclosure;

[0036] Figure 4b is a structural schematic diagram of a table area detection model provided according to an embodiment of the present disclosure;

[0037] Figure 5 is a flowchart of a method for training a table recognition model provided according to an embodiment of the present disclosure;

[0038] Figure 6 is a flowchart of another method for training a table recognition model provided according to an embodiment of the present disclosure;

[0039] Figure 7 is a flowchart of another method for training a table recognition model provided according to an embodiment of the present disclosure;

[0040] Figure 8 is a structural schematic diagram of a table recognition device provided according to an embodiment of the present disclosure;

[0041] Fig. 9 It is a structural schematic diagram of a training device for a table recognition model provided according to an embodiment of the present disclosure;

[0042] Fig.10 It is a block diagram of an electronic device used to implement the table recognition method or the table recognition model training method of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0043] Figure 1a 1 is a flow chart of a table recognition method provided according to an embodiment of the present disclosure. The method is applicable to the case of recognizing a table structure in an image. The method can be executed by a table recognition device, which can be implemented in software and / or hardware and can be integrated into an electronic device. Figure 1a As shown, the table recognition method of this embodiment may include:

[0044] S101, expanding the table size of the table image to be recognized according to a preset total number of rows and a total number of columns to obtain an expanded target table image; the total number of rows and the total number of columns are both fixed values;

[0045] S102, performing table recognition on the target table image to obtain table attribute information of the target table image; the table attribute information includes row validity information, column validity information, coordinates of table vertices, and connectivity between adjacent table vertices;

[0046] S103, reconstructing the table according to the table attribute information of the target table image to obtain the cell information in the table image to be identified.

[0047] In the embodiment of the present disclosure, the table image to be identified may belong to table scenes such as tilted, distorted, with merged cells, empty cells, etc. The original row dividing lines and column dividing lines in the table image to be identified are respectively used as valid rows and valid columns, and the original table vertices in the table image to be identified are used as valid vertices, for example Figure 1b The table in includes 4 valid rows, 3 valid columns, and 10 valid vertices.

[0048] In order to facilitate the recognition of tables of different sizes, the table images to be recognized in different table scenarios can be expanded into a table structure of M×N columns. M and N are the total number of rows and columns, respectively, both of which are preset fixed values ​​and can be set based on experience. Figure 1b and Figure 1c By expanding the table image to be recognized with a size of 4×3, a target table image with a size of M×N can be obtained. The target table image includes not only valid vertices 11, valid rows 12, and valid columns 13, but also invalid vertices 21, invalid rows 22, and invalid columns 23 introduced by the expansion. In order to improve the versatility of table recognition, M and N can be 80 and 30 respectively. It should be noted that in the process of table expansion, a fixed row spacing and column spacing can be used, and invalid rows and invalid columns are introduced into the table image to be recognized to obtain the expanded target table image.

[0049] In the disclosed embodiment, the table structure is decoupled into the following three attributes: validity information of rows and columns in the table (collectively referred to as row / column validity information), coordinates of table vertices and connectivity between adjacent table vertices; wherein, row / column validity information is used to characterize whether the rows / columns in the table are valid; the coordinates of table vertices include the horizontal and vertical coordinates of the table vertices; and the connectivity between adjacent table vertices is used to characterize whether adjacent table vertices are connected.

[0050] Specifically, straight line detection may be performed on the table image to be identified, and the original table size in the table image to be identified may be obtained according to the straight line detection result. The original table size may be expanded using M and N to obtain the expanded target table image.

[0051] Based on the three attributes of the table, the table structure of the target table image is recognized to obtain the table attribute information of the target table image; wherein the table attribute information includes the validity information of the row / column, the coordinates of the table vertices and the connectivity between the adjacent table vertices; Figure 1c Taking the target table image shown as an example, the table attribute information may include the validity of each row, the validity of each column, the coordinates of each table vertex, and the connectivity between adjacent table vertices.

[0052] In the process of table reconstruction, table vertices on invalid rows and invalid columns can be filtered out, and only table vertices on valid rows and valid columns are used for table reconstruction; table vertices on valid rows and valid columns are screened according to the connectivity between adjacent table vertices to distinguish valid vertices from invalid vertices; and cell information in the table image to be identified is obtained by combining the continuous relationship between valid vertices. It should be noted that the cell information in the table image to be identified can include both the logical information of the cells in the table image to be identified, such as the row number and the column number, and the pixel information of the cells, such as the vertex coordinates of the cells.

[0053] The technical solution provided by the disclosed embodiment expands the table image to be identified to a fixed size to obtain an expanded target table image, decouples the table into three table attributes, namely, validity information of rows / columns, coordinates of table vertices, and connectivity between adjacent table vertices, and identifies the target table image based on the above table attributes to obtain table attribute information of the target table image, and reconstructs cells in the target table image according to the table attribute information of the target table image to obtain cell information in the table image to be identified, which not only improves the accuracy of table recognition, but is also applicable to different table scenarios and has universality.

[0054] Figure 2a is a flow chart of another table recognition method provided according to an embodiment of the present disclosure. Figure 2a The table recognition method of this embodiment may include:

[0055] S201, expanding the table size of the table image to be recognized according to a preset total number of rows and a total number of columns to obtain an expanded target table image; the total number of rows and the total number of columns are both fixed values;

[0056] S202, extracting features from each row and each column in the target table image respectively to obtain row features of each row and column features of each column in the target table image;

[0057] S203, inputting the row features of each row into a row decoder to obtain a decoding result of each row, and inputting the column features of each column into a column decoder in a table recognition model to obtain a decoding result of each column;

[0058] S204, determining row attribute information of each row and column attribute information of each column in the target table image according to the decoding results of each row and the decoding results of each column, as table attribute information of the target table image;

[0059] S205, reconstructing the table according to the table attribute information of the target table image to obtain the cell information in the table image to be identified.

[0060] In the disclosed embodiment, the row attribute information may include the validity of the row, the ordinate of the row, and the connectivity between the table vertices in adjacent rows; the column attribute information may include the validity of the column, the abscissa of the column, and the connectivity between the table vertices in adjacent columns. According to the ordinate of each row and the abscissa of each column, the abscissa and ordinate of each table vertex in the target table image may be obtained. By respectively determining the row attribute information of each row and the column attribute information of each column in the target table image as the table attribute information of the target table image, that is, by granulating the table attribute information into row attribute information and column attribute information, the accuracy of the table attribute information is improved, thereby being able to improve the accuracy of subsequent table reconstruction based on the table attribute information.

[0061] Figure 2b is a structural diagram of a table recognition model provided according to an embodiment of the present disclosure, referring to Figure 2b , the table recognition model may include a feature extractor, a row decoder and a column decoder. In the table recognition process, the feature extractor may be used to extract features of each row and each column in the target table image, respectively, to obtain row features of each row and column features of each column in the target table image; the row features of each row are input into the row decoder to obtain the decoding results of each row, and the column features of each column are input into the column decoder to obtain the decoding results of each column.

[0062] Among them, the feature extractor can use ResNet50 (convolutional neural network) as the backbone network, and several cascaded Transformer Encoder (transform decoder) networks can be set after the backbone network to enable a global receptive field. In the feature extraction process, the backbone network is used to process the target table image to obtain 5 downsampled feature maps of different resolutions, from which the top 3 downsampled feature maps with confidence are selected and input into the cascaded Transformer Encoder network to obtain the row features of each row and the column features of each column in the target table image. Among them, the column decoder and the row decoder can both use decoders with deformable attention mechanisms. Compared with the original attention mechanism, the deformable attention mechanism uses reference points and sampling points to make each query (query vector) correspond to a reference point, and the query only interacts with the sampling points, which greatly saves the amount of calculation, and the positions of the reference points and sampling points are learnable.

[0063] In addition, the validity of each row, the vertical coordinate of each row and the connectivity between table vertices in adjacent rows can be determined based on the decoding results of each row; and the validity of each column, the horizontal coordinate of each column and the connectivity between table vertices in adjacent columns can be determined based on the decoding results of each column.

[0064] Exemplarily, the table recognition model may also include a row output layer and a column output layer, both of which may adopt a FFN (Fully Connected Network) structure, and the network parameters of the two may be shared. Further, the row output layer may include a validity subclassifier, a connectivity subclassifier, and a subregressor, the validity subclassifier and the connectivity subclassifier may both be binary classifications, the validity subclassifier is used to determine whether the corresponding row is valid, the connectivity subclassifier is used to determine whether the table vertices in adjacent rows are connected, and the subregressor is used to determine the ordinate of the corresponding row. Similarly, the column output layer may include a validity subclassifier, a connectivity subclassifier, and a subregressor, the validity subclassifier is used to determine whether the corresponding column is valid, the connectivity subclassifier is used to determine whether the table vertices in adjacent rows are connected, and the subregressor is used to determine the abscissa of the corresponding column.

[0065] In an optional embodiment, the row attribute information of each row and the column attribute information of each column in the target table image are determined respectively according to the decoding results of each row and the decoding results of each column, including: using a row enhancement decoder, based on a cross-attention mechanism, using the decoding results of each row as a key vector and a key-value vector, and using the decoding results of each column as a query vector, to obtain an enhanced decoding result of each row; using a column enhancement decoder, based on a cross-attention mechanism, using the decoding results of each column as a key vector and a key-value vector, and using the decoding results of each row as a query vector, to obtain an enhanced decoding result of each column; determining the row attribute information of each row in the target table image according to the enhanced decoding results of each row, and determining the column attribute information of each column in the target table image according to the enhanced decoding results of each column.

[0066] Figure 2c is a schematic diagram of another table recognition model provided according to an embodiment of the present disclosure, referring to Figure 2cIn addition to the feature extractor, row decoder and column decoder, the table recognition model also includes a row enhancement decoder and a column enhancement decoder. Through the row enhancement decoder, based on the cross attention mechanism, the decoding results of each row are used as the key vector and key value vector, and the decoding results of each column are used as the query vector to obtain the enhanced decoding results of each row; through the column enhancement decoder, based on the cross attention mechanism, the decoding results of each column are used as the key vector and key value vector, and the decoding results of each row are used as the query vector to obtain the enhanced decoding results of each column. In other words, the row enhancement decoder and the column enhancement decoder use the cross attention mechanism to enhance the decoding results of the row and the decoding results of the column respectively, and the enhanced features are directly added to the original features to obtain the fused coding features after the output. The self-attention module used by the row enhancement decoder and the column enhancement decoder may include a self-attention layer, a multi-head attention layer and a forward feedback network, and the output dimension size is the same as the input dimension size.

[0067] By introducing row enhancement decoder and column enhancement decoder into the table recognition model, the decoding results of rows and columns are enhanced respectively based on the cross-attention mechanism, which can further improve the accuracy of table recognition.

[0068] The technical solution provided by the embodiment of the present disclosure improves the accuracy of table recognition by proposing a new table recognition structure based on three types of attribute information of the table; and further improves the accuracy of table recognition by introducing a row enhancement decoder and a column enhancement decoder in the table recognition structure.

[0069] Figure 3 is a flowchart of another table recognition method provided according to an embodiment of the present disclosure, which is used to further define table reconstruction. Figure 3 The table recognition method of this embodiment may include:

[0070] S301, expanding the table size of the table image to be recognized according to a preset total number of rows and a total number of columns to obtain an expanded target table image; the total number of rows and the total number of columns are both fixed values;

[0071] S302, performing table recognition on the target table image to obtain table attribute information of the target table image; the table attribute information includes row validity information, column validity information, coordinates of table vertices, and connectivity between adjacent table vertices;

[0072] S303, reconstructing the table according to the table attribute information of the target table image to obtain the cell information in the table image to be identified;

[0073] S304, for a starting vertex of any cell in the target table image, obtaining a first table vertex in the same row as the starting vertex and in the next column, and a second table vertex in the same column as the first table vertex and in the next row; determining a vertex of the cell according to connectivity between the first table vertex and the second table vertex;

[0074] S305, for the starting vertex of any cell in the target table image, obtain the third table vertex that is in the same column as the starting vertex and located in the next row, and the fourth table vertex that is in the same column as the third table vertex and located in the next column; determine the vertex of the cell according to the connectivity between the third table vertex and the fourth table vertex.

[0075] In the process of table reconstruction, the vertex in the upper left corner of the target table image can be used as the starting point, and each table vertex can be traversed in sequence along a fixed direction to obtain the cell information to which the table vertex belongs. The fixed direction can be from left to right or from top to bottom. The starting vertex of a cell can be the vertex in the upper left corner of the cell.

[0076] For the starting vertex of any cell in the target table image, increase the width by 1 in the right direction to obtain the first table vertex in the same row as the starting vertex and in the next column, and increase the width by 1 in the downward direction from the first table vertex to obtain the second table vertex in the same column as the first table vertex and in the next row; if the connectivity between the first table vertex and the second table vertex is connected, the first table vertex is the vertex of the cell; if the connectivity between the first table vertex and the second table vertex is not connected, continue to increase the width by 1 in the right direction to obtain a new first table vertex in the same row as the first table vertex and in the next column, and increase the width by 1 in the downward direction from the new first table vertex to obtain a new second table vertex in the same column as the new first table vertex and in the next row, until the connectivity between the new first table vertex and the new second table vertex is connected, then the new first table vertex is used as the table vertex of the cell.

[0077] For the starting vertex of any cell in the target table image, the width is increased by 1 downward to obtain the third table vertex in the same column and the next row as the starting vertex, and the width is increased by 1 to the right from the third table vertex to obtain the fourth table vertex in the same column and the next column as the third table vertex; if the connectivity between the third table vertex and the fourth table vertex is connected, the third table vertex is the vertex of the cell; if the connectivity between the third table vertex and the fourth table vertex is not connected, the width is continued to be increased by 1 downward to obtain a new third table vertex in the same column and the next row as the third table vertex, and the width is increased by 1 to the right from the new third table vertex to obtain a new fourth table vertex in the same column and the next column as the new third table vertex, until the connectivity between the new third table vertex and the new fourth table vertex is connected, then the new third table vertex is used as the table vertex of the cell. The logical information of the cell can be obtained by using the starting vertex of the cell, the cell vertex in the same column as the starting vertex, and the cell vertex in the same column as the starting vertex. In addition, the pixel information of the cell can also be obtained according to the coordinates of each vertex in the cell.

[0078] By reconstructing the target table image based on the three attribute information of the table, we can obtain the index number of the starting row / column of the cell in the table image to be identified, the vertex coordinates of the cell, etc., which can be used to automatically identify the content in the cell.

[0079] The technical solution provided by the embodiment of the present disclosure also provides an adaptive table reconstruction method for the three attribute information of the table. The cell information in the table image to be identified can be obtained according to the table attribute information of the target table image, that is, the automatic recognition of the table structure in the table image to be identified is realized based on the three attribute information of the table.

[0080] In an optional embodiment, the validity information of the row is the validity classification information of the corresponding row, and the validity information of the column is the validity classification information of the corresponding column; before reconstructing the table according to the table attribute information of the target table image, it also includes: determining the valid rows and valid columns in the target table image according to the validity classification information of the corresponding rows and the validity classification information of the corresponding columns; sorting each valid row according to the vertical coordinates of the valid rows in the target table image, and sorting each valid column according to the horizontal coordinates of the valid columns in the target table image.

[0081] In the embodiment of the present disclosure, the value of the validity classification information can be 1 or 0, 1 is used to indicate that the row is valid, and 0 is used to indicate that the row is invalid. When the size of the target table image is M×N, the dimension of the row validity classification information of the target table image is M, where the row with a value of 1 is a valid row, and the row with a value of 0 is an invalid row; and the dimension of the column validity classification information of the target table image is N, where the column with a value of 1 is a valid column, and the column with a value of 0 is an invalid column.

[0082] After obtaining each valid row and each valid column in the target table image, each valid row is sorted according to the ordinate of the valid row, and each valid column is sorted according to the abscissa of the valid column, so as to obtain the row index number of each valid row and the column index number of each valid column. By determining the row index number of the valid row and the column index number of the valid column in the target table image during the table recognition process, the index numbers of the valid rows and the valid columns are made random, so as to further improve the recognition accuracy of large-size tables.

[0083] In an optional implementation, the row validity information is the number of valid rows, and the column validity information is the number of valid columns.

[0084] During the table recognition process, only the valid number of rows and columns in the target table image can be output, so that the index numbers of the valid rows and columns are fixed. Taking the table size of the table image to be recognized as 4×5 as an example, the number of valid rows in the target table image is 4, and the number of valid columns is 5, that is, the first 4 rows in the target table image are valid rows, and the first 5 columns are valid columns. By fixing the index numbers of the valid rows and columns, the table recognition process can be simplified and the table recognition efficiency can be improved.

[0085] Figure 4a is a flow chart of another table recognition method provided according to an embodiment of the present disclosure. This embodiment is an optional solution proposed based on the above embodiment. Figure 4a The table recognition method of this embodiment may include:

[0086] S401, using at least two detection units cascaded in the table area detection model to sequentially perform table area detection on the image to be recognized; wherein the intersection-over-union ratio thresholds used by each detection unit in the cascade are sequentially increased;

[0087] S402, determining a table area detection result of the image to be recognized according to the table area output by the detection unit at the tail, as the table image to be recognized;

[0088] S403, expanding the table size of the table image to be recognized according to a preset total number of rows and a total number of columns to obtain an expanded target table image; the total number of rows and the total number of columns are both fixed values;

[0089] S404, performing table recognition on the target table image to obtain table attribute information of the target table image; the table attribute information includes row validity information, column validity information, coordinates of table vertices, and connectivity between adjacent table vertices;

[0090] S405 , reconstructing the table according to the table attribute information of the target table image to obtain the cell information in the table image to be identified.

[0091] Among them, the table area detection model is used to detect the table area in the image to be identified to obtain the table image to be identified, that is, to extract the table area image from the image to be identified as the table image to be identified. The table area detection model may include at least two cascaded detection units, that is, the output of the previous detection unit can be used as the input of the next detection unit. Taking the adjacent first detection unit and the second detection unit as an example, the output of the first detection unit can be used as the input of the second detection unit. The intersection over union (IOU) thresholds adopted by each cascaded detection unit increase successively, that is, the IOU threshold adopted by the previous detection unit is lower than the IOU threshold adopted by the next detection unit. Still taking the adjacent first detection unit and the second detection unit as an example, the IOU threshold adopted by the first detection unit is lower than the IOU threshold adopted by the second detection unit. Therefore, the IOU threshold adopted by the detection unit located at the tail is the largest.

[0092] Each cascaded detection unit adopts a corresponding IOU threshold to perform table area detection on the image to be identified, and determines the table image to be identified according to the table area output by the detection unit at the tail. That is to say, adopting the maximum IOU threshold to determine the table image to be identified can improve the accuracy of table area detection and avoid misidentifying content outside the table area as the table area; moreover, gradually increasing the IOU threshold through the cascaded detection units can also improve the stability of table area detection and avoid overfitting.

[0093] In an optional embodiment, the table area detection model includes at least a feature extraction unit, a basic detection unit and at least one auxiliary detection unit; the auxiliary detection unit is correspondingly provided with a pooling unit; the at least two detection units in the table area detection model are used to perform table area detection on the image to be identified in turn, including: using the feature extraction unit to extract features from the image to be identified to obtain a feature map to be identified; using the basic detection unit to perform table area detection on the feature map to be identified to obtain a basic table area; using a correspondingly set pooling unit to scale the feature map corresponding to the basic table area to obtain a scaled feature map; using the auxiliary detection unit to perform table area detection on the scaled feature map.

[0094] refer to Figure 4b , the detection unit located at the head can be used as the basic detection unit, and the detection units other than the basic detection unit can be used as auxiliary detection units. The basic detection unit and the auxiliary detection unit can obtain the basic table and the auxiliary table respectively; the auxiliary detection unit also outputs the confidence of the auxiliary table. There is no specific limit on the number of auxiliary detection units. For example, refer to Figure 4b , three auxiliary detection units can be set, where the IOU threshold used by the basic detection unit is 0.4, and the IOU thresholds used by the three auxiliary detection units are 0.5, 0.6, and 0.7 respectively.

[0095] refer to Figure 4b ,The auxiliary detection unit is also correspondingly provided with a pooling unit for unifying the feature maps corresponding to the table area into a fixed size. Specifically, the image to be identified is input into the feature extraction unit to obtain a feature map to be identified, and the basic detection unit is used to perform table area detection on the feature map to be identified to obtain a basic table area; the feature map corresponding to the basic table area is determined, and the feature map corresponding to the basic table area is input into the pooling unit corresponding to the first auxiliary detection unit, and the feature map is unified into a fixed size and then passed into the first auxiliary detection unit to obtain the first table area output by the first auxiliary detection unit, and the confidence of the first table area; the feature map corresponding to the first table area is determined, and the feature map corresponding to the first table area is input into the pooling unit corresponding to the second auxiliary detection unit, and the feature map is unified into a fixed size and then passed into the second auxiliary detection unit to obtain the second table area output by the second auxiliary detection unit, and the confidence of the second table area; the feature map corresponding to the second table area is determined, and the feature map corresponding to the second table area is input into the pooling unit corresponding to the third auxiliary detection unit, and the feature map is unified into a fixed size and then passed into the third auxiliary detection unit to obtain the third table area output by the third auxiliary detection unit, and the confidence of the third table area, select the third table area whose confidence is greater than the IOU threshold adopted by the third auxiliary detection unit, and determine the result image to be identified according to the selected third table area.

[0096] It should be noted that the disclosed embodiments do not specifically limit the network structures of the feature extraction unit, the table detection unit, and the pooling unit. The network structure of the table area detection model can improve the accuracy and stability of the table image to be identified, thereby improving the accuracy of subsequent table recognition of cells in the table image to be identified.

[0097] The technical solution provided by the disclosed embodiment uses each detection unit cascaded in the table area detection model to perform table area detection on the image to be identified using the corresponding IOU threshold, and determines the table image to be identified based on the table area output by the detection unit at the tail, which can not only improve the accuracy of the table image to be identified, but also avoid overfitting.

[0098] Figure 5 1 is a flow chart of a method for training a table recognition model according to an embodiment of the present disclosure. The method is applicable to training a table recognition model. The method can be executed by a training device for a table recognition model, which can be implemented in software and / or hardware and can be integrated into an electronic device. Figure 5 As shown, the image recognition training method of this embodiment may include:

[0099] S501, expanding the table size of the sample table image according to a preset total number of rows and a total number of columns to obtain an expanded auxiliary table image; the total number of rows and the total number of columns are both fixed values;

[0100] S502, using the table recognition model to be trained to perform table recognition on the auxiliary table image to obtain table attribute information of the auxiliary table image; the table attribute information of the auxiliary table image includes validity information of predicted rows, validity information of predicted columns, coordinates of predicted table vertices, and connectivity between adjacent predicted table vertices in the auxiliary table image;

[0101] S503, training the table recognition model according to the table attribute information marked by the sample table image and the table attribute information of the auxiliary table image; the table attribute information marked by the sample table image includes the validity information of the marked rows in the sample table image, the validity information of the marked columns, the coordinates of the marked table vertices and the connectivity between adjacent marked table vertices.

[0102] The original row dividing lines and dividing lines in the sample table image are used as valid rows and valid columns, respectively, and the original table vertices in the sample table image are used as valid vertices. In order to facilitate the recognition of tables of different sizes, the sample table images in different table scenes can be expanded into a table structure of M×N columns. M and N are the total number of rows and the total number of columns, respectively, both of which are preset fixed values ​​and can be set according to experience.

[0103] In the disclosed embodiment, the table structure is decoupled into the following three attributes: validity information of rows / columns in the table, coordinates of table vertices, and connectivity between adjacent table vertices. Specifically, straight line detection can be performed on the sample table image, and the original table size in the table image to be identified can be obtained according to the straight line detection result, and the original table size can be expanded using M and N to obtain the expanded auxiliary table image.

[0104] For ease of expression, the rows, columns, and table vertices in the auxiliary table image can be referred to as predicted rows, predicted columns, and predicted table vertices, respectively; the rows, columns, and table vertices in the sample table image can be referred to as labeled rows, labeled columns, and labeled table vertices, respectively.

[0105] Based on the three attributes of the table, the table structure of the auxiliary table image is recognized to obtain the validity of each predicted row, the validity of each predicted column, the coordinates of each predicted table vertex, and the connectivity between adjacent predicted table vertices in the auxiliary table image. According to the table attribute information annotated by the sample table image, the validity of each annotated row, the validity of each annotated column, the coordinates of each annotated table vertex, and the connectivity between adjacent annotated table vertices in the sample table image are obtained. In the training process of the table recognition model, a loss function can be constructed based on the validity of each predicted row, the validity of each predicted column, the coordinates of each predicted table vertex and the connectivity between adjacent predicted table vertices in the auxiliary table image, as well as the validity of each annotated row, the validity of each annotated column, the coordinates of each annotated table vertex and the connectivity between adjacent annotated table vertices in the sample table image, and the loss function is used to train the table recognition model.

[0106] The technical solution provided by the disclosed embodiment expands the sample table image to a fixed size to obtain an expanded auxiliary table image, decouples the table into three table attributes: validity information of rows / columns, coordinates of table vertices, and connectivity between adjacent table vertices, and recognizes the auxiliary table image based on the above table attributes to obtain table attribute information of the auxiliary table image. A loss function is constructed by combining the table attribute information marked by the sample table image and the table attribute information of the auxiliary table image, and the loss function is used to train a table recognition model, which not only improves the performance of the table recognition model, but also makes the table recognition model applicable to different table scenarios and has universality.

[0107] In an optional embodiment, the table recognition model to be trained includes a feature extractor, a row decoder and a column decoder; the table recognition model to be trained is used to perform table recognition on the auxiliary table image to obtain table attribute information of the auxiliary table image, including: using the feature extractor to perform feature extraction on each predicted row and each predicted column in the auxiliary table image, respectively, to obtain row features of each predicted row and column features of each predicted column in the auxiliary table image; inputting the row features of each predicted row into the row decoder to obtain a decoding result of each predicted row, and inputting the column features of each predicted column into the column decoder to obtain a decoding result of each column; and according to the decoding results of each predicted row and the decoding results of each predicted column, determining the row attribute information of each predicted row and the column attribute information of each predicted column in the auxiliary table image, respectively, as the table attribute information of the auxiliary table image.

[0108] Among them, the row attribute information of the predicted row may include the validity of the predicted row, the ordinate of the predicted row and the connectivity between the vertices of the predicted table in the adjacent predicted rows; the column attribute information of the predicted column may include the validity of the predicted column, the abscissa of the predicted column and the connectivity between the vertices of the predicted table in the adjacent predicted columns. According to the ordinate of each predicted row and the abscissa of each predicted column, the abscissa and ordinate of each predicted table vertex in the auxiliary table image can be obtained. By respectively determining the row attribute information of each predicted row and the column attribute information of each predicted column in the auxiliary table image as the table attribute information of the auxiliary table image, that is, by fine-graining the table attribute information into row attribute information and column attribute information, the accuracy of the table attribute information is improved. It should be noted that the row attribute information of the marked row in the sample table image may include the validity of the marked row, the ordinate of the marked row and the connectivity between the vertices of the marked table in the adjacent marked rows; the column attribute information of the marked column may include the validity of the marked column, the abscissa of the marked column and the connectivity between the vertices of the marked table in the adjacent marked columns.

[0109] The table recognition model may include a feature extractor, a row decoder and a column decoder. In the table recognition process, the feature extractor may be used to extract features of each predicted row and each predicted column in the auxiliary table image, respectively, to obtain row features of each predicted row and column features of each predicted column in the auxiliary table image; the row features of each predicted row are input into the row decoder to obtain the decoding results of each predicted row, and the column features of each predicted column are input into the column decoder to obtain the decoding results of each predicted column.

[0110] In addition, the validity of each prediction row, the vertical coordinate of each prediction row, and the connectivity between the vertices of the prediction table in adjacent prediction rows can be determined based on the decoding results of each prediction row; and the validity of each prediction column, the horizontal coordinate of each prediction column, and the connectivity between the vertices of the prediction table in adjacent prediction columns can be determined based on the decoding results of each prediction column.

[0111] In an optional embodiment, the table recognition model to be trained also includes a row enhancement decoder and a column enhancement decoder; the row attribute information of each predicted row and the column attribute information of each predicted column in the auxiliary table image are determined according to the decoding results of each predicted row and the decoding results of each predicted column, respectively, including: through the row enhancement decoder, based on the cross-attention mechanism, the decoding results of each predicted row are used as the key vector and the key-value vector, and the decoding results of each predicted column are used as the query vector to obtain the enhanced decoding results of each predicted row; through the column enhancement decoder, based on the cross-attention mechanism, the decoding results of each predicted column are used as the key vector and the key-value vector, and the decoding results of each predicted row are used as the query vector to obtain the enhanced decoding results of each predicted column; the row attribute information of each predicted row in the auxiliary table image is determined according to the enhanced decoding results of each predicted row, and the column attribute information of each predicted column in the auxiliary table image is determined according to the enhanced decoding results of each predicted column.

[0112] In addition to the feature extractor, row decoder and column decoder, the table recognition model also includes a row enhancement decoder and a column enhancement decoder. Through the row enhancement decoder, based on the cross attention mechanism, the decoding results of each predicted row are used as the key vector and key value vector, and the decoding results of each predicted column are used as the query vector to obtain the enhanced decoding results of each predicted row; through the column enhancement decoder, based on the cross attention mechanism, the decoding results of each predicted column are used as the key vector and key value vector, and the decoding results of each predicted row are used as the query vector to obtain the enhanced decoding results of each predicted column. In other words, the row enhancement decoder and the column enhancement decoder use the cross attention mechanism to enhance the decoding results of the predicted row and the decoding results of the predicted column, respectively. By introducing the row enhancement decoder and the column enhancement decoder in the table recognition model, and enhancing the decoding results of the predicted row and the decoding results of the predicted column, respectively, based on the cross attention mechanism, the accuracy of table recognition can be further improved.

[0113] In an optional implementation, the validity information of the predicted rows is the number of valid predicted rows, and the validity information of the predicted columns is the number of valid predicted columns.

[0114] During the table recognition process, only the valid number of rows and columns in the auxiliary table image can be output, so that the index numbers of the valid rows and columns are fixed. Taking the table size of the sample table image as 4×5 as an example, the number of valid rows in the auxiliary table image is 4, and the number of valid columns is 5, that is, the first 4 rows in the auxiliary table image are valid rows, and the first 5 columns are valid columns. By fixing the index numbers of the valid rows and columns, the table recognition process can be simplified and the table recognition efficiency can be improved.

[0115] Figure 6 FIG. 1 is a flowchart of another method for training a table recognition model according to an embodiment of the present disclosure. Figure 6 , the training method of the table recognition model of this embodiment may include:

[0116] S601, expanding the table size of the sample table image according to a preset total number of rows and a total number of columns to obtain an expanded auxiliary table image; the total number of rows and the total number of columns are both fixed values;

[0117] S602, using the table recognition model to be trained to perform table recognition on the auxiliary table image to obtain table attribute information of the auxiliary table image; the table attribute information of the auxiliary table image includes validity information of predicted rows, validity information of predicted columns, coordinates of predicted table vertices, and connectivity between adjacent predicted table vertices in the auxiliary table image;

[0118] The sample table image has annotated table attribute information, and the annotated table attribute information includes validity information of annotated rows and columns in the sample table image, coordinates of annotated table vertices, and connectivity between adjacent annotated table vertices.

[0119] S603, determining the annotated valid rows in the sample table image according to the validity information of the annotated rows in the sample table image, and determining the annotated valid columns in the sample table image according to the validity information of the annotated columns in the sample table image;

[0120] S604, selecting a matching predicted valid row for the marked valid row from the predicted rows in the auxiliary table image, and selecting a matching predicted valid column for the marked valid column from the predicted columns in the auxiliary table image, to obtain a candidate matching relationship;

[0121] S605, selecting a target matching relationship from the candidate matching relationships according to the table attribute information of the auxiliary table image and the predicted valid rows and predicted valid columns in the candidate matching relationships;

[0122] The target matching relationship includes a matching relationship between predicted rows in the auxiliary table image and annotated rows in the sample table image, and a matching relationship between predicted columns in the auxiliary table image and annotated columns in the sample table image;

[0123] S606, constructing a loss function according to the target matching relationship, the table attribute information of the auxiliary table image and the table attribute information annotated by the sample table image, and using the loss function to train the table recognition model.

[0124] In the auxiliary table image, the validity information of the predicted row may be the validity classification information of the predicted row, and the validity information of the predicted column may be the validity classification information of the predicted column. The value of the validity classification information may be 1 or 0, 1 is used to indicate that the predicted row is valid, and 0 is used to indicate that the predicted row is invalid. When the size of the auxiliary table image is M×N, the dimension of the row validity classification information of the auxiliary table image is M, where the rows with a value of 1 are valid rows, and the rows with a value of 0 are invalid predicted rows; and the dimension of the column validity classification information of the auxiliary table image is N, where the columns with a value of 1 are valid columns, and the columns with a value of 0 are invalid columns. The validity information of the annotated rows in the sample table image may be the number of valid annotated rows and the number of valid annotated columns. Taking the sample table image as an example in which the number of valid annotated rows is 4 and the number of valid annotated columns is 5, the first 4 annotated rows in the sample table image are all valid annotated rows, and the remaining annotated rows are invalid annotated rows; the first 5 annotated columns are all valid annotated columns, and the remaining annotated columns are invalid annotated columns.

[0125] In the training process of the table recognition model, for each valid annotated row in the sample table image, a matching predicted valid row can be randomly selected from each predicted row in the auxiliary table image based on the Hungarian matching algorithm; and for each valid annotated column in the sample table image, a matching predicted valid column can be randomly selected from each predicted column in the auxiliary table image based on the Hungarian matching algorithm to obtain a candidate matching relationship. Still taking the sample table image as having 4 valid annotated rows, 5 valid annotated columns, and the auxiliary table image having a size of 80×30 as an example, the i-th predicted row, j-th predicted row, k-th predicted row, and l-th predicted row in the auxiliary table image can be used as the predicted valid row for matching the first valid annotated row, the second valid annotated row, the third valid annotated row, and the fourth valid annotated row in the sample table image, respectively, to obtain a candidate matching relationship; wherein i, j, k, and l are all natural numbers, and have different values. The process of selecting a matching predicted valid column for a labeled valid column is similar to the process of selecting a matching predicted valid row for a labeled valid row, which will not be described in detail.

[0126] For each candidate matching relationship, the row attribute information of the predicted valid row and the column attribute information of the predicted valid column in the candidate matching relationship can be determined according to the table attribute information of the auxiliary table image, and the candidate matching relationships can be screened according to the row attribute information of the predicted valid row and the column attribute information of the predicted valid column in the candidate matching relationship to obtain the target matching relationship, and the matching relationship between the predicted valid row in the auxiliary table image and the annotated valid row in the sample table image, and the matching relationship between the predicted valid column in the auxiliary table image and the annotated valid column in the sample table image can be obtained according to the target matching relationship, that is, the annotated row matched by each predicted row in the auxiliary table image and the label column matched by each predicted column in the auxiliary table image are obtained. According to the annotated row matched by each predicted row in the auxiliary table image and the label column matched by each predicted column in the auxiliary table image, the loss function of the table recognition model can be constructed, and the table recognition model can be trained using the loss function.

[0127] During the training process of the table recognition model, by randomly constructing multiple candidate matching relationships and selecting target matching relationships from the multiple candidate matching relationships, the index numbers of the predicted valid rows and predicted valid columns in the auxiliary table image are made random, thereby further improving the recognition accuracy of large-size tables.

[0128] In an optional embodiment, the target matching relationship is selected from the candidate matching relationships according to the table attribute information of the auxiliary table image, and the predicted valid rows and predicted valid columns in the candidate matching relationships, including: determining the ordinate of the predicted valid row, the validity classification confidence of the predicted valid row, the abscissa of the predicted valid column, and the validity classification confidence of the predicted valid column in the candidate matching relationship according to the table attribute information of the auxiliary table image; determining the regression distance of the predicted valid row according to the ordinate of the predicted valid row and the ordinate of the center point in the auxiliary table image; and determining the regression distance of the predicted valid column according to the abscissa of the predicted valid column and the abscissa of the center point in the auxiliary table image; determining the classification distance of the predicted valid row according to the validity classification confidence of the predicted valid row, and determining the classification distance of the predicted valid column according to the validity classification confidence of the predicted valid column; determining the matching distance of the candidate matching relationship according to the regression distance and classification distance of the predicted valid row, and the regression distance and classification distance of the predicted valid column, and selecting the candidate matching relationship with the smallest matching distance as the target matching relationship.

[0129] In the disclosed embodiment, the validity information of the predicted row may include whether the predicted row is valid, the validity classification confidence, the ordinate of the predicted row, and the connectivity between adjacent prediction table vertices in the predicted row; the validity information of the predicted column may include whether the predicted column is valid, the validity classification confidence, the abscissa of the predicted column, and the connectivity between adjacent prediction table vertices in the predicted column. The validity classification confidence has a value range of 0 to 1, and the greater the validity classification confidence, the higher the probability that the corresponding predicted row and predicted column are valid.

[0130] Exemplarily, the regression distance of the predicted effective row is determined according to the L1 distance between the ordinate of the predicted effective row and the ordinate of the center point in the auxiliary table image; and the regression distance of the predicted effective column is determined according to the L1 distance between the abscissa of the predicted effective column and the abscissa of the center point in the auxiliary table image; the classification distance of the predicted effective row is determined according to the validity classification confidence of the predicted effective row, and the classification distance of the predicted effective column is determined according to the validity classification confidence of the predicted effective column; the regression distance, classification distance of the predicted effective row, the regression distance and classification distance of the predicted effective column are accumulated to obtain the matching distance of the candidate matching relationship, and the candidate matching relationship with the smallest matching distance is selected as the target matching relationship. Among them, the L1 distance can also be called the mean absolute error distance.

[0131] By randomly constructing candidate matching relationships, determining the matching distance of each candidate matching relationship, and taking the candidate matching relationship with the smallest matching distance as the target matching relationship, the table recognition model can also learn to predict the index numbers of valid rows and columns during training, thereby improving the recognition accuracy of large-size tables.

[0132] In an optional embodiment, a loss function is constructed according to the target matching relationship, the table attribute information of the auxiliary table image and the table attribute information annotated by the sample table image, and the loss function is used to train the table recognition model, including: determining the validity classification loss based on the target matching relationship, according to the validity information of the predicted rows in the auxiliary table image and the validity information of the corresponding annotated rows in the sample table image, and according to the validity information of the predicted columns in the auxiliary table image and the validity information of the corresponding annotated columns in the sample table image; determining the coordinate regression loss based on the target matching relationship, according to the coordinates of the predicted table vertices in the auxiliary table image and the coordinates of the corresponding annotated table vertices in the sample table image; determining the connectivity classification loss based on the target matching relationship, according to the connectivity between adjacent predicted table vertices in the auxiliary table image and the connectivity between corresponding adjacent predicted table vertices in the sample table image; and training the table recognition model based on the validity classification loss, the coordinate regression loss and the connectivity classification loss.

[0133] Among them, the loss function of the table recognition model may include the following three types of losses: validity classification loss of predicted rows / predicted columns, coordinate regression loss of predicted table vertices, and connectivity classification loss between adjacent predicted table vertices; the above three types of losses are determined separately, and the above three types of losses are accumulated to obtain the loss function of the table recognition model, and the loss function is used to train the table recognition model.

[0134] The technical solution provided by the embodiment of the present disclosure randomly constructs multiple candidate matching relationships and selects a target matching relationship from the multiple candidate matching relationships, so that the index numbers of the predicted valid rows and predicted valid columns in the auxiliary table image are random, thereby further improving the recognition accuracy of large-size tables.

[0135] Figure 7 FIG. 1 is a flowchart of another method for training a table recognition model according to an embodiment of the present disclosure. Figure 7 , the training method of the table recognition model of this embodiment may include:

[0136] S701, using at least two detection units cascaded in the table area detection model to perform table area detection on the sample image in sequence; wherein the intersection-over-union ratio thresholds used by each detection unit in the cascade are increased in sequence;

[0137] S702, determining a table area detection result of the sample image according to the table area output by the detection unit at the tail as the sample table image;

[0138] S703, expanding the table size of the sample table image according to a preset total number of rows and a total number of columns to obtain an expanded auxiliary table image; the total number of rows and the total number of columns are both fixed values;

[0139] S704, using the table recognition model to be trained to perform table recognition on the auxiliary table image to obtain table attribute information of the auxiliary table image; the table attribute information of the auxiliary table image includes validity information of predicted rows, validity information of predicted columns, coordinates of predicted table vertices, and connectivity between adjacent predicted table vertices in the auxiliary table image;

[0140] S705, training the table recognition model according to the table attribute information marked by the sample table image and the table attribute information of the auxiliary table image; the table attribute information marked by the sample table image includes the validity information of the marked rows in the sample table image, the validity information of the marked columns, the coordinates of the marked table vertices and the connectivity between adjacent marked table vertices.

[0141] The table area detection model is also used to detect the table area in the sample image to obtain a sample table image. The table area detection model may include at least two cascaded detection units, that is, the output of the previous detection unit can be used as the input of the next detection unit. The intersection over union (IOU) thresholds used by each cascaded detection unit increase in sequence. Therefore, the IOU threshold used by the detection unit at the tail is the largest.

[0142] The corresponding IOU threshold is used by each cascaded detection unit to perform table area detection on the sample image, and the sample table image is determined according to the table area output by the detection unit at the tail. That is to say, using the maximum IOU threshold to determine the sample table image can improve the accuracy of table area detection; moreover, by gradually increasing the IOU threshold through the cascaded detection units, the stability of table area detection can also be improved to avoid overfitting.

[0143] In an optional embodiment, the table recognition model to be trained includes a feature extractor, a row decoder and a column decoder; the table recognition model to be trained is used to perform table recognition on the auxiliary table image to obtain table attribute information of the auxiliary table image, including: using the feature extractor to perform feature extraction on each predicted row and each predicted column in the auxiliary table image, respectively, to obtain row features of each predicted row and column features of each predicted column in the auxiliary table image; inputting the row features of each predicted row into the row decoder to obtain a decoding result of each predicted row, and inputting the column features of each predicted column into the column decoder to obtain a decoding result of each column; and according to the decoding results of each predicted row and the decoding results of each predicted column, determining the row attribute information of each predicted row and the column attribute information of each predicted column in the auxiliary table image, respectively, as the table attribute information of the auxiliary table image.

[0144] In the disclosed embodiment, the detection unit located at the head can be used as the basic detection unit, and the detection units other than the basic detection unit can be used as auxiliary detection units. The auxiliary detection unit is also provided with a corresponding pooling unit for unifying the feature map corresponding to the table area into a fixed size. Through the network structure of the above table area detection model, the accuracy and stability of the sample table image can be improved, thereby improving the accuracy of the subsequent table recognition of the cells in the sample table image.

[0145] The technical solution provided by the embodiment of the present disclosure performs table area detection on the sample image by using the corresponding IOU threshold through each detection unit cascaded in the table area detection model, and determines the sample table image according to the table area output by the detection unit at the tail, which can not only improve the accuracy of the sample table image, but also avoid overfitting.

[0146] Figure 8 1 is a schematic diagram of a table recognition device according to an embodiment of the present disclosure. This embodiment is suitable for performing a task of recognizing cells in a table. The device can be implemented in software and / or hardware. Figure 8 As shown, the table recognition device 800 of this embodiment may include:

[0147] The target table expansion module 810 is used to expand the table size of the table image to be recognized according to a preset total number of rows and a total number of columns to obtain an expanded target table image; the total number of rows and the total number of columns are both fixed values;

[0148] A target table recognition module 820 is used to perform table recognition on the target table image to obtain table attribute information of the target table image; the table attribute information includes row validity information, column validity information, coordinates of table vertices and connectivity between adjacent table vertices;

[0149] The target table reconstruction module 830 is used to reconstruct the table according to the table attribute information of the target table image to obtain the cell information in the table image to be identified.

[0150] In an optional implementation, the target table identification module 820 includes:

[0151] A target feature extraction unit, used to extract features from each row and each column in the target table image, respectively, to obtain row features of each row and column features of each column in the target table image;

[0152] A target feature decoding unit, used for inputting the row feature of each row into a row decoder to obtain a decoding result of each row, and inputting the column feature of each column into a column decoder in a table recognition model to obtain a decoding result of each column;

[0153] The target table recognition unit is used to determine the row attribute information of each row and the column attribute information of each column in the target table image according to the decoding results of each row and the decoding results of each column, as the table attribute information of the target table image.

[0154] In an optional implementation, the target table identification unit includes:

[0155] A row enhancement subunit is used to obtain enhanced decoding results of each row through a row enhancement decoder based on a cross attention mechanism, using the decoding results of each row as a key vector and a key value vector, and using the decoding results of each column as a query vector;

[0156] The column enhancement subunit is used to obtain the enhanced decoding results of each column through the column enhancement decoder based on the cross attention mechanism, taking the decoding results of each column as the key vector and the key value vector, and taking the decoding results of each row as the query vector;

[0157] The target table identification subunit is used to determine the row attribute information of each row in the target table image according to the enhanced decoding results of each row, and to determine the column attribute information of each column in the target table image according to the enhanced decoding results of each column.

[0158] In an optional implementation, the row validity information is the number of valid rows, and the column validity information is the number of valid columns.

[0159] In an optional implementation manner, the row validity information is validity classification information of the corresponding row, and the column validity information is validity classification information of the corresponding column;

[0160] The table recognition device 800 further includes an effective sorting module, and the effective sorting module includes:

[0161] The valid row and column unit is used to determine the valid rows and valid columns in the target table image according to the validity classification information of the corresponding rows and the validity classification information of the corresponding columns;

[0162] The effective sorting unit is used to sort the effective rows according to the vertical coordinates of the effective rows in the target table image, and to sort the effective columns according to the horizontal coordinates of the effective columns in the target table image.

[0163] In an optional implementation, the target table reconstruction module 830 includes:

[0164] A first cell reconstruction unit is used to obtain, for a starting vertex of any cell in a target table image, a first table vertex in the same row as the starting vertex and located in the next column, and a second table vertex in the same column as the first table vertex and located in the next row; and determine the vertex of the cell according to the connectivity between the first table vertex and the second table vertex;

[0165] The second cell reconstruction unit is used to obtain, for the starting vertex of any cell in the target table image, a third table vertex that is in the same column as the starting vertex and located in the next row, and a fourth table vertex that is in the same column as the third table vertex and located in the next column; and determine the vertex of the cell according to the connectivity between the third table vertex and the fourth table vertex.

[0166] In an optional implementation, the table recognition device 800 further includes a table area detection module, and the table area detection module includes:

[0167] A cascade detection unit, used to sequentially perform table area detection on the image to be identified using at least two cascaded detection units in the table area detection model; wherein the intersection-over-union ratio thresholds used by each cascaded detection unit are sequentially increased;

[0168] The table area detection unit is used to determine the table area detection result of the image to be identified according to the table area output by the detection unit at the tail, as the table image to be identified.

[0169] In an optional implementation, the table area detection model includes at least a feature extraction unit, a basic detection unit and at least one auxiliary detection unit; the auxiliary detection unit is correspondingly provided with a pooling unit;

[0170] The cascade detection unit is specifically used for:

[0171] Using the feature extraction unit, extracting features from the image to be identified to obtain a feature map to be identified;

[0172] Using the basic detection unit to perform table area detection on the feature map to be identified to obtain a basic table area;

[0173] Scaling the feature map corresponding to the base table area using a correspondingly set pooling unit to obtain a scaled feature map;

[0174] The auxiliary detection unit is used to perform table area detection on the scaled feature map.

[0175] The above table recognition model can be used as a separate table structure recognition product, or the table structure recognition capability can be embedded in the document service to obtain a complete text recognition product. For example, a table area detection model can be pre-installed in the table recognition model, and an OCR (Optical Character Recognition) content recognition module can be post-installed to form an end-to-end text recognition product. Compared with related solutions, this solution can cover more table data scenarios, reduce model deployment and model size, and has obvious advantages over previous methods.

[0176] The technical solution of the disclosed embodiment decouples the table structure into three table attributes, and provides table structure recognition based on the three table attributes, which can solve structural recognition problems such as tilted, curved, blank cells, and wireless tables. It can extract table data from financial statements, company bills, test paper tables, etc., which can greatly simplify manpower and improve office work efficiency.

[0177] Fig. 9 1 is a schematic diagram of a structure of a training device for a table recognition model provided according to an embodiment of the present disclosure. This embodiment is applicable to training a table recognition model. The device can be implemented in software and / or hardware. Fig. 9 As shown, the table recognition model training device 900 of this embodiment may include:

[0178] The auxiliary table expansion module 910 expands the table size of the sample table image according to a preset total number of rows and a total number of columns to obtain an expanded auxiliary table image; the total number of rows and the total number of columns are both fixed values;

[0179] The auxiliary table recognition module 920 uses the table recognition model to be trained to perform table recognition on the auxiliary table image to obtain table attribute information of the auxiliary table image; the table attribute information of the auxiliary table image includes validity information of predicted rows, validity information of predicted columns, coordinates of predicted table vertices, and connectivity between adjacent predicted table vertices in the auxiliary table image;

[0180] The table model training module 930 is used to train the table recognition model according to the table attribute information marked by the sample table image and the table attribute information of the auxiliary table image; the table attribute information marked by the sample table image includes the validity information of the marked rows in the sample table image, the validity information of the marked columns, the coordinates of the marked table vertices and the connectivity between adjacent marked table vertices.

[0181] In an optional implementation, the table recognition model to be trained includes a feature extractor, a row decoder, and a column decoder;

[0182] The auxiliary table identification module 920 includes:

[0183] An auxiliary feature extraction unit, used to use the feature extractor to extract features from each predicted row and each predicted column in the auxiliary table image, respectively, to obtain row features of each predicted row and column features of each predicted column in the auxiliary table image;

[0184] an auxiliary feature decoding unit, configured to input the row feature of each predicted row into the row decoder to obtain a decoding result of each predicted row, and input the column feature of each predicted column into the column decoder to obtain a decoding result of each column;

[0185] The auxiliary table identification unit is used to determine the row attribute information of each prediction row and the column attribute information of each prediction column in the auxiliary table image according to the decoding results of each prediction row and the decoding results of each prediction column, as the table attribute information of the auxiliary table image.

[0186] In an optional implementation, the table recognition model to be trained further includes a row enhancement decoder and a column enhancement decoder;

[0187] The auxiliary table recognition unit comprises:

[0188] A row enhancement subunit, configured to obtain an enhanced decoding result of each predicted row by using the row enhancement decoder based on a cross attention mechanism, taking the decoding result of each predicted row as a key vector and a key value vector, and taking the decoding result of each predicted column as a query vector;

[0189] A column enhancement subunit is used to obtain an enhanced decoding result of each prediction column through a column enhancement decoder based on a cross attention mechanism, using the decoding result of each prediction column as a key vector and a key value vector, and using the decoding result of each prediction row as a query vector;

[0190] The auxiliary table identification subunit is used to determine the row attribute information of each predicted row in the auxiliary table image according to the enhanced decoding results of each predicted row, and to determine the column attribute information of each predicted column in the auxiliary table image according to the enhanced decoding results of each predicted column.

[0191] In an optional implementation, the validity information of the predicted rows is the number of valid predicted rows, and the validity information of the predicted columns is the number of valid predicted columns.

[0192] In an optional implementation, the table model training module 930 includes:

[0193] A marking validity unit, used to determine the marking valid rows in the sample table image according to the validity information of the marking rows in the sample table image, and to determine the marking valid columns in the sample table image according to the validity information of the marking columns in the sample table image;

[0194] A candidate matching unit, configured to select a matching predicted valid row for the marked valid row from the predicted rows in the auxiliary table image, and select a matching predicted valid column for the marked valid column from the predicted columns in the auxiliary table image, to obtain a candidate matching relationship;

[0195] a target matching unit, configured to select a target matching relationship from the candidate matching relationships according to the table attribute information of the auxiliary table image, and the predicted valid rows and predicted valid columns in the candidate matching relationships; the target matching relationship includes a matching relationship between the predicted rows in the auxiliary table image and the annotated rows in the sample table image, and a matching relationship between the predicted columns in the auxiliary table image and the annotated columns in the sample table image;

[0196] The table model training unit is used to construct a loss function according to the target matching relationship, the table attribute information of the auxiliary table image and the table attribute information marked by the sample table image, and use the loss function to train the table recognition model.

[0197] In an optional implementation, the target matching unit includes:

[0198] A prediction attribute subunit, used to determine the ordinate of the predicted valid row, the validity classification confidence of the predicted valid row, the abscissa of the predicted valid column and the validity classification confidence of the predicted valid column in the candidate matching relationship according to the table attribute information of the auxiliary table image;

[0199] A regression distance subunit, for determining the regression distance of the predicted valid row according to the ordinate of the predicted valid row and the ordinate of the center point in the auxiliary table image; and for determining the regression distance of the predicted valid column according to the abscissa of the predicted valid column and the abscissa of the center point in the auxiliary table image;

[0200] A classification distance subunit, used to determine the classification distance of the predicted valid rows according to the validity classification confidence of the predicted valid rows, and to determine the classification distance of the predicted valid columns according to the validity classification confidence of the predicted valid columns;

[0201] The target matching subunit is used to determine the matching distance of the candidate matching relationship according to the regression distance and classification distance of the predicted valid rows and the regression distance and classification distance of the predicted valid columns, and select the candidate matching relationship with the smallest matching distance as the target matching relationship.

[0202] In an optional implementation, the table model training unit includes:

[0203] A validity classification loss subunit, for determining the validity classification loss based on the target matching relationship, according to the validity information of the predicted rows in the auxiliary table image and the validity information of the corresponding annotated rows in the sample table image, and according to the validity information of the predicted columns in the auxiliary table image and the validity information of the corresponding annotated columns in the sample table image;

[0204] A coordinate regression loss subunit, for determining a coordinate regression loss based on the target matching relationship according to the coordinates of the predicted table vertices in the auxiliary table image and the coordinates of the corresponding annotated table vertices in the sample table image;

[0205] A connectivity classification loss subunit, for determining a connectivity classification loss based on the target matching relationship and according to connectivity between adjacent predicted table vertices in the auxiliary table image and connectivity between corresponding adjacent predicted table vertices in the sample table image;

[0206] The table model training subunit is used to train the table recognition model according to the validity classification loss, the coordinate regression loss and the connectivity classification loss.

[0207] In an optional implementation, the table recognition model training device 900 further includes a table area detection module, and the table area detection module includes:

[0208] A cascade detection unit, used to sequentially perform table area detection on a sample image using at least two cascaded detection units in a table area detection model; wherein the intersection-over-union ratio thresholds used by each cascaded detection unit are sequentially increased;

[0209] The table area detection unit is used to determine the table area detection result of the sample image according to the table area output by the detection unit at the tail as the sample table image.

[0210] In an optional implementation, the table area detection model includes at least a feature extraction unit, a basic detection unit and at least one auxiliary detection unit; the auxiliary detection unit is correspondingly provided with a pooling unit;

[0211] The cascade detection unit is specifically used for:

[0212] Using the feature extraction unit, extracting features from the sample image to obtain a sample feature map;

[0213] Using the basic detection unit to perform table area detection on the sample feature map to obtain a basic table area;

[0214] Scaling the feature map corresponding to the base table area using a correspondingly set pooling unit to obtain a scaled feature map;

[0215] The auxiliary detection unit is used to perform table area detection on the scaled feature map.

[0216] The technical solution of the disclosed embodiment decouples the table structure into three table attributes, and provides table structure recognition based on the three table attributes, which can solve structural recognition problems such as tilted, curved, blank cells, and wireless tables. It can extract table data from financial statements, company bills, test paper tables, etc., which can greatly simplify manpower and improve office work efficiency.

[0217] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0218] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0219] Fig.10 It is a block diagram of an electronic device used to implement the table recognition method or the table recognition model training method of the embodiment of the present disclosure.

[0220] Fig.10 A schematic block diagram of an example electronic device 1000 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0221] like Fig.10As shown, the electronic device 1000 includes a computing unit 1001, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1002 or a computer program loaded from a storage unit 1008 into a random access memory (RAM) 1003. In the RAM 1003, various programs and data required for the operation of the electronic device 1000 can also be stored. The computing unit 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0222] Multiple components in the electronic device 1000 are connected to the I / O interface 1005, including: an input unit 1006, such as a keyboard, a mouse, etc.; an output unit 1007, such as various types of displays, speakers, etc.; a storage unit 1008, such as a disk, an optical disk, etc.; and a communication unit 1009, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1009 allows the electronic device 1000 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0223] The computing unit 1001 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1001 performs the various methods and processes described above, such as a table recognition method or a training method for a table recognition model. For example, in some embodiments, the table recognition method or the training method for a table recognition model may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 1008. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 1000 via the ROM 1002 and / or the communication unit 1009. When the computer program is loaded into the RAM 1003 and executed by the computing unit 1001, one or more steps of the table recognition method or the training method for the table recognition model described above may be performed. Alternatively, in other embodiments, the computing unit 1001 may be configured to execute the table recognition method or the table recognition model training method in any other appropriate manner (for example, by means of firmware).

[0224] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0225] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0226] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0227] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0228] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0229] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0230] Artificial intelligence is a discipline that studies how to use computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, planning, etc.). It includes both hardware-level and software-level technologies. Artificial intelligence hardware technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, and big data processing; artificial intelligence software technologies mainly include computer vision technology, speech recognition technology, natural language processing technology, machine learning / deep learning technology, big data processing technology, knowledge graph technology, and other major directions.

[0231] Cloud computing refers to a technology system that uses network access to elastically scalable shared physical or virtual resource pools. Resources can include servers, operating systems, networks, software, applications, and storage devices, and can be deployed and managed on demand and in a self-service manner. Cloud computing technology can provide efficient and powerful data processing capabilities for technical applications such as artificial intelligence and blockchain, as well as model training.

[0232] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0233] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A table recognition method, include: Expanding the table size of the table image to be recognized according to the preset total number of rows and total number of columns to obtain an expanded target table image; The total number of rows and the total number of columns are both fixed values; Performing table recognition on the target table image to obtain table attribute information of the target table image; the table attribute information includes row validity information, column validity information, coordinates of table vertices and connectivity between adjacent table vertices; the row validity information is validity classification information of the corresponding row, and the column validity information is validity classification information of the corresponding column; The connectivity between the adjacent table vertices is used to indicate whether the adjacent table vertices are connected; Reconstructing the table according to the table attribute information of the target table image to obtain the cell information in the table image to be identified, including: screening the table vertices on the valid rows and valid columns according to the connectivity between the adjacent table vertices in the table attribute information of the target table image to distinguish the valid vertices from the invalid vertices; combining the connectivity between the valid vertices to obtain the cell information in the table image to be identified; Wherein, before reconstructing the table according to the table attribute information of the target table image, the method further includes: Determine valid rows and valid columns in the target table image according to validity classification information of corresponding rows and validity classification information of corresponding columns; The valid rows are sorted according to the ordinates of the valid rows in the target table image, and the valid columns are sorted according to the abscissas of the valid columns in the target table image.

2. The method according to claim 1, in, The performing table recognition on the target table image to obtain table attribute information of the target table image includes: Extracting features from each row and each column in the target table image respectively to obtain row features of each row and column features of each column in the target table image; Input the row features of each row into the row decoder to obtain the decoding results of each row, and input the column features of each column into the column decoder in the table recognition model to obtain the decoding results of each column; According to the decoding results of the rows and the decoding results of the columns, row attribute information of the rows and column attribute information of the columns in the target table image are determined respectively as table attribute information of the target table image.

3. The method according to claim 2, in, The step of determining row attribute information of each row and column attribute information of each column in the target table image according to the decoding results of each row and the decoding results of each column respectively includes: Through the row-enhanced decoder, based on the cross-attention mechanism, the decoding results of each row are used as the key vector and key value vector, and the decoding results of each column are used as the query vector to obtain the enhanced decoding results of each row; Through the column-enhanced decoder, based on the cross-attention mechanism, the decoding results of each column are used as the key vector and key value vector, and the decoding results of each row are used as the query vector to obtain the enhanced decoding results of each column; The row attribute information of each row in the target table image is determined according to the enhanced decoding results of each row, and the column attribute information of each column in the target table image is determined according to the enhanced decoding results of each column.

4. The method according to claim 1, in, The row validity information is the number of valid rows, and the column validity information is the number of valid columns.

5. The method according to any one of claims 1 to 4, in, The step of reconstructing the table according to the table attribute information of the target table image to obtain the cell information in the table image to be identified includes: For a starting vertex of any cell in a target table image, obtain a first table vertex in the same row as the starting vertex and in the next column, and a second table vertex in the same column as the first table vertex and in the next row; determine the vertex of the cell according to the connectivity between the first table vertex and the second table vertex; For the starting vertex of any cell in the target table image, obtain the third table vertex that is in the same column as the starting vertex and in the next row, and the fourth table vertex that is in the same row as the third table vertex and in the next column; determine the vertex of the cell based on the connectivity between the third table vertex and the fourth table vertex.

6. The method according to any one of claims 1 to 4, further comprising: include: At least two detection units cascaded in the table area detection model are used to perform table area detection on the image to be identified in sequence; wherein the intersection-over-union ratio thresholds used by each detection unit in the cascade are increased in sequence; The table area detection result of the image to be recognized is determined according to the table area output by the detection unit located at the tail, as the table image to be recognized.

7. The method according to claim 6, in, The table area detection model at least includes a feature extraction unit, a basic detection unit and at least one auxiliary detection unit; the auxiliary detection unit is correspondingly provided with a pooling unit; The method of using at least two detection units in the table area detection model to sequentially perform table area detection on the image to be recognized includes: Using the feature extraction unit, extracting features from the image to be identified to obtain a feature map to be identified; Using the basic detection unit to perform table area detection on the feature map to be identified to obtain a basic table area; Scaling the feature map corresponding to the base table area using a correspondingly set pooling unit to obtain a scaled feature map; The auxiliary detection unit is used to perform table area detection on the scaled feature map.

8. A training method for a table recognition model, include: Expanding the table size of the sample table image according to the preset total number of rows and total number of columns to obtain an expanded auxiliary table image; The total number of rows and the total number of columns are both fixed values; Using the table recognition model to be trained to perform table recognition on the auxiliary table image, and obtain table attribute information of the auxiliary table image; the table attribute information of the auxiliary table image includes validity information of predicted rows, validity information of predicted columns, coordinates of predicted table vertices and connectivity between adjacent predicted table vertices in the auxiliary table image; the connectivity between adjacent predicted table vertices is used to characterize whether the adjacent predicted table vertices are connected; The table recognition model is trained according to the table attribute information annotated by the sample table image and the table attribute information of the auxiliary table image; the table attribute information annotated by the sample table image includes validity information of the annotated rows, validity information of the annotated columns, coordinates of the annotated table vertices and connectivity between adjacent annotated table vertices in the sample table image; the connectivity between adjacent annotated table vertices is used to characterize whether the adjacent annotated table vertices are connected; The table recognition model is used to determine table attribute information in implementing the table recognition method as described in any one of claims 1 to 7.

9. The method according to claim 8, in, The table recognition model to be trained includes a feature extractor, a row decoder and a column decoder; The table recognition model to be trained is used to perform table recognition on the auxiliary table image to obtain table attribute information of the auxiliary table image, including: The feature extractor is used to extract features of each predicted row and each predicted column in the auxiliary table image, respectively, to obtain row features of each predicted row and column features of each predicted column in the auxiliary table image; Inputting the row features of each predicted row into the row decoder to obtain a decoding result of each predicted row, and inputting the column features of each predicted column into the column decoder to obtain a decoding result of each column; According to the decoding results of the predicted rows and the decoding results of the predicted columns, the row attribute information of the predicted rows and the column attribute information of the predicted columns in the auxiliary table image are respectively determined as the table attribute information of the auxiliary table image.

10. The method according to claim 9, in, The table recognition model to be trained also includes a row enhancement decoder and a column enhancement decoder; The step of determining row attribute information of each predicted row and column attribute information of each predicted column in the auxiliary table image according to the decoding results of each predicted row and the decoding results of each predicted column respectively includes: By using the row enhancement decoder, based on the cross attention mechanism, the decoding results of each predicted row are used as the key vector and the key value vector, and the decoding results of each predicted column are used as the query vector to obtain the enhanced decoding results of each predicted row; Through the column enhancement decoder, based on the cross attention mechanism, the decoding results of each predicted column are used as the key vector and key value vector, and the decoding results of each predicted row are used as the query vector to obtain the enhanced decoding results of each predicted column; The row attribute information of each predicted row in the auxiliary table image is determined according to the enhanced decoding result of each predicted row, and the column attribute information of each predicted column in the auxiliary table image is determined according to the enhanced decoding result of each predicted column.

11. The method according to claim 8, in, The validity information of the prediction row is the number of valid prediction rows, and the validity information of the prediction column is the number of valid prediction columns.

12. The method according to claim 8, in, The step of training the table recognition model according to the table attribute information marked by the sample table image and the table attribute information of the auxiliary table image comprises: Determine the annotated valid rows in the sample table image according to the validity information of the annotated rows in the sample table image, and determine the annotated valid columns in the sample table image according to the validity information of the annotated columns in the sample table image; Selecting a matching predicted valid row for the marked valid row from the predicted rows in the auxiliary table image, and selecting a matching predicted valid column for the marked valid column from the predicted columns in the auxiliary table image, to obtain a candidate matching relationship; According to the table attribute information of the auxiliary table image, and the predicted valid rows and predicted valid columns in the candidate matching relationships, a target matching relationship is selected from the candidate matching relationships; the target matching relationship includes a matching relationship between the predicted rows in the auxiliary table image and the annotated rows in the sample table image, and a matching relationship between the predicted columns in the auxiliary table image and the annotated columns in the sample table image; A loss function is constructed based on the target matching relationship, the table attribute information of the auxiliary table image and the table attribute information annotated by the sample table image, and the table recognition model is trained using the loss function.

13. The method according to claim 12, in, The selecting a target matching relationship from the candidate matching relationships according to the table attribute information of the auxiliary table image and the predicted valid rows and predicted valid columns in the candidate matching relationships comprises: Determine, according to the table attribute information of the auxiliary table image, the ordinate of the predicted valid row, the validity classification confidence of the predicted valid row, the abscissa of the predicted valid column, and the validity classification confidence of the predicted valid column in the candidate matching relationship; Determine the regression distance of the predicted valid row according to the ordinate of the predicted valid row and the ordinate of the center point in the auxiliary table image; and determine the regression distance of the predicted valid column according to the abscissa of the predicted valid column and the abscissa of the center point in the auxiliary table image; Determine the classification distance of the predicted valid rows according to the validity classification confidence of the predicted valid rows, and determine the classification distance of the predicted valid columns according to the validity classification confidence of the predicted valid columns; According to the regression distance and classification distance of the predicted valid rows and the regression distance and classification distance of the predicted valid columns, the matching distance of the candidate matching relationship is determined, and the candidate matching relationship with the smallest matching distance is selected as the target matching relationship.

14. The method according to claim 12, in, The step of constructing a loss function based on the target matching relationship, the table attribute information of the auxiliary table image, and the table attribute information annotated by the sample table image, and using the loss function to train the table recognition model includes: Based on the target matching relationship, determining the validity classification loss according to the validity information of the predicted rows in the auxiliary table image and the validity information of the corresponding annotated rows in the sample table image, and according to the validity information of the predicted columns in the auxiliary table image and the validity information of the corresponding annotated columns in the sample table image; Based on the target matching relationship, determining the coordinate regression loss according to the coordinates of the predicted table vertices in the auxiliary table image and the coordinates of the corresponding annotated table vertices in the sample table image; Determining a connectivity classification loss based on the target matching relationship and the connectivity between adjacent predicted table vertices in the auxiliary table image and the connectivity between corresponding adjacent predicted table vertices in the sample table image; The table recognition model is trained according to the validity classification loss, the coordinate regression loss and the connectivity classification loss.

15. The method according to any one of claims 8 to 14, before expanding the table size of the sample table image according to the preset total number of rows and total number of columns, further comprising: include: At least two detection units cascaded in the table area detection model are used to perform table area detection on the sample image in sequence; wherein the intersection-over-union ratio thresholds used by each detection unit in the cascade are increased in sequence; The table area detection result of the sample image is determined according to the table area output by the detection unit located at the tail, as the sample table image.

16. The method according to claim 15, in, The table area detection model at least includes a feature extraction unit, a basic detection unit and at least one auxiliary detection unit; the auxiliary detection unit is correspondingly provided with a pooling unit; The method of sequentially performing table area detection on the sample image by using at least two detection units cascaded in the table area detection model includes: Using the feature extraction unit, extracting features from the sample image to obtain a sample feature map; Using the basic detection unit to perform table area detection on the sample feature map to obtain a basic table area; Scaling the feature map corresponding to the base table area using a correspondingly set pooling unit to obtain a scaled feature map; The auxiliary detection unit is used to perform table area detection on the scaled feature map.

17. A table recognition device, include: A target table expansion module is used to expand the table size of the table image to be recognized according to a preset total number of rows and total number of columns to obtain an expanded target table image; The total number of rows and the total number of columns are both fixed values; A target table recognition module is used to perform table recognition on the target table image to obtain table attribute information of the target table image; the table attribute information includes row validity information, column validity information, coordinates of table vertices and connectivity between adjacent table vertices; the row validity information is validity classification information of the corresponding row, and the column validity information is validity classification information of the corresponding column; The connectivity between the adjacent table vertices is used to indicate whether the adjacent table vertices are connected; A target table reconstruction module, used to reconstruct the table according to the table attribute information of the target table image, and obtain the cell information in the table image to be identified; The target table reconstruction module is specifically used to: screen the table vertices on valid rows and valid columns according to the connectivity between the adjacent table vertices in the table attribute information of the target table image to distinguish valid vertices from invalid vertices; and obtain the cell information in the table image to be identified by combining the connectivity between the valid vertices; Wherein, the table recognition device further includes an effective sorting module, and the effective sorting module includes: The valid row and column unit is used to determine the valid rows and valid columns in the target table image according to the validity classification information of the corresponding rows and the validity classification information of the corresponding columns; The effective sorting unit is used to sort the effective rows according to the vertical coordinates of the effective rows in the target table image, and to sort the effective columns according to the horizontal coordinates of the effective columns in the target table image.

18. The device according to claim 17, in, The target table recognition module includes: A target feature extraction unit, used to extract features from each row and each column in the target table image, respectively, to obtain row features of each row and column features of each column in the target table image; A target feature decoding unit, used for inputting the row feature of each row into a row decoder to obtain a decoding result of each row, and inputting the column feature of each column into a column decoder in a table recognition model to obtain a decoding result of each column; The target table recognition unit is used to determine the row attribute information of each row and the column attribute information of each column in the target table image according to the decoding results of each row and the decoding results of each column, as the table attribute information of the target table image.

19. The device according to claim 18, in, The target table identification unit comprises: A row enhancement subunit is used to obtain enhanced decoding results of each row through a row enhancement decoder based on a cross attention mechanism, using the decoding results of each row as a key vector and a key value vector, and using the decoding results of each column as a query vector; The column enhancement subunit is used to obtain the enhanced decoding results of each column through the column enhancement decoder based on the cross attention mechanism, taking the decoding results of each column as the key vector and the key value vector, and taking the decoding results of each row as the query vector; The target table identification subunit is used to determine the row attribute information of each row in the target table image according to the enhanced decoding results of each row, and to determine the column attribute information of each column in the target table image according to the enhanced decoding results of each column.

20. The device according to claim 17, in, The row validity information is the number of valid rows, and the column validity information is the number of valid columns.

21. The device according to any one of claims 17 to 20, in, The target table reconstruction module includes: A first cell reconstruction unit is used to obtain, for a starting vertex of any cell in a target table image, a first table vertex in the same row as the starting vertex and located in the next column, and a second table vertex in the same column as the first table vertex and located in the next row; and determine the vertex of the cell according to the connectivity between the first table vertex and the second table vertex; The second cell reconstruction unit is used to obtain, for the starting vertex of any cell in the target table image, a third table vertex that is in the same column as the starting vertex and located in the next row, and a fourth table vertex that is in the same column as the third table vertex and located in the next column; and determine the vertex of the cell according to the connectivity between the third table vertex and the fourth table vertex.

22. The device according to any one of claims 17 to 20, wherein the table recognition device further comprises a table area detection module, wherein the table area detection module include: A cascade detection unit, used to sequentially perform table area detection on the image to be identified using at least two cascaded detection units in the table area detection model; wherein the intersection-over-union ratio thresholds used by each cascaded detection unit are sequentially increased; The table area detection unit is used to determine the table area detection result of the image to be identified according to the table area output by the detection unit at the tail, as the table image to be identified.

23. The device according to claim 22, in, The table area detection model at least includes a feature extraction unit, a basic detection unit and at least one auxiliary detection unit; the auxiliary detection unit is correspondingly provided with a pooling unit; The cascade detection unit is specifically used for: Using the feature extraction unit, extracting features from the image to be identified to obtain a feature map to be identified; Using the basic detection unit to perform table area detection on the feature map to be identified to obtain a basic table area; Scaling the feature map corresponding to the base table area using a correspondingly set pooling unit to obtain a scaled feature map; The auxiliary detection unit is used to perform table area detection on the scaled feature map.

24. A training device for a table recognition model, include: The auxiliary table expansion module expands the table size of the sample table image according to the preset total number of rows and total number of columns to obtain an expanded auxiliary table image; The total number of rows and the total number of columns are both fixed values; an auxiliary table recognition module, which uses a table recognition model to be trained to perform table recognition on the auxiliary table image to obtain table attribute information of the auxiliary table image; the table attribute information of the auxiliary table image includes validity information of predicted rows, validity information of predicted columns, coordinates of predicted table vertices, and connectivity between adjacent predicted table vertices in the auxiliary table image; the connectivity between adjacent predicted table vertices is used to characterize whether the adjacent predicted table vertices are connected; A table model training module, used for training the table recognition model according to the table attribute information annotated by the sample table image and the table attribute information of the auxiliary table image; the table attribute information annotated by the sample table image includes the validity information of the annotated rows, the validity information of the annotated columns, the coordinates of the annotated table vertices and the connectivity between adjacent annotated table vertices in the sample table image; the connectivity between adjacent annotated table vertices is used to characterize whether the adjacent annotated table vertices are connected; The table recognition model is used to determine table attribute information in implementing the table recognition method as described in any one of claims 1 to 7.

25. The device according to claim 24, in, The table recognition model to be trained includes a feature extractor, a row decoder and a column decoder; The auxiliary table recognition module includes: An auxiliary feature extraction unit, used to use the feature extractor to extract features from each predicted row and each predicted column in the auxiliary table image, respectively, to obtain row features of each predicted row and column features of each predicted column in the auxiliary table image; an auxiliary feature decoding unit, configured to input the row feature of each predicted row into the row decoder to obtain a decoding result of each predicted row, and input the column feature of each predicted column into the column decoder to obtain a decoding result of each column; The auxiliary table identification unit is used to determine the row attribute information of each prediction row and the column attribute information of each prediction column in the auxiliary table image according to the decoding results of each prediction row and the decoding results of each prediction column, as the table attribute information of the auxiliary table image.

26. The device according to claim 25, in, The table recognition model to be trained also includes a row enhancement decoder and a column enhancement decoder; The auxiliary table recognition unit comprises: A row enhancement subunit, configured to obtain an enhanced decoding result of each predicted row by using the row enhancement decoder based on a cross attention mechanism, taking the decoding result of each predicted row as a key vector and a key value vector, and taking the decoding result of each predicted column as a query vector; A column enhancement subunit is used to obtain an enhanced decoding result of each prediction column through a column enhancement decoder based on a cross attention mechanism, using the decoding result of each prediction column as a key vector and a key value vector, and using the decoding result of each prediction row as a query vector; The auxiliary table identification subunit is used to determine the row attribute information of each predicted row in the auxiliary table image according to the enhanced decoding results of each predicted row, and to determine the column attribute information of each predicted column in the auxiliary table image according to the enhanced decoding results of each predicted column.

27. The device according to claim 24, in, The validity information of the prediction row is the number of valid prediction rows, and the validity information of the prediction column is the number of valid prediction columns.

28. The device according to claim 24, in, The table model training module includes: A marking validity unit, used to determine the marking valid rows in the sample table image according to the validity information of the marking rows in the sample table image, and to determine the marking valid columns in the sample table image according to the validity information of the marking columns in the sample table image; A candidate matching unit, configured to select a matching predicted valid row for the marked valid row from the predicted rows in the auxiliary table image, and select a matching predicted valid column for the marked valid column from the predicted columns in the auxiliary table image, to obtain a candidate matching relationship; a target matching unit, configured to select a target matching relationship from the candidate matching relationships according to the table attribute information of the auxiliary table image, and the predicted valid rows and predicted valid columns in the candidate matching relationships; the target matching relationship includes a matching relationship between the predicted rows in the auxiliary table image and the annotated rows in the sample table image, and a matching relationship between the predicted columns in the auxiliary table image and the annotated columns in the sample table image; The table model training unit is used to construct a loss function according to the target matching relationship, the table attribute information of the auxiliary table image and the table attribute information marked by the sample table image, and use the loss function to train the table recognition model.

29. The device according to claim 28, in, The target matching unit comprises: A prediction attribute subunit, used to determine the ordinate of the predicted valid row, the validity classification confidence of the predicted valid row, the abscissa of the predicted valid column and the validity classification confidence of the predicted valid column in the candidate matching relationship according to the table attribute information of the auxiliary table image; A regression distance subunit, for determining the regression distance of the predicted valid row according to the ordinate of the predicted valid row and the ordinate of the center point in the auxiliary table image; and for determining the regression distance of the predicted valid column according to the abscissa of the predicted valid column and the abscissa of the center point in the auxiliary table image; A classification distance subunit, used to determine the classification distance of the predicted valid rows according to the validity classification confidence of the predicted valid rows, and to determine the classification distance of the predicted valid columns according to the validity classification confidence of the predicted valid columns; The target matching subunit is used to determine the matching distance of the candidate matching relationship according to the regression distance and classification distance of the predicted valid rows and the regression distance and classification distance of the predicted valid columns, and select the candidate matching relationship with the smallest matching distance as the target matching relationship.

30. The device according to claim 28, in, The table model training unit includes: A validity classification loss subunit, for determining the validity classification loss based on the target matching relationship, according to the validity information of the predicted rows in the auxiliary table image and the validity information of the corresponding annotated rows in the sample table image, and according to the validity information of the predicted columns in the auxiliary table image and the validity information of the corresponding annotated columns in the sample table image; A coordinate regression loss subunit, for determining a coordinate regression loss based on the target matching relationship according to the coordinates of the predicted table vertices in the auxiliary table image and the coordinates of the corresponding annotated table vertices in the sample table image; A connectivity classification loss subunit, for determining a connectivity classification loss based on the target matching relationship and according to connectivity between adjacent predicted table vertices in the auxiliary table image and connectivity between corresponding adjacent predicted table vertices in the sample table image; The table model training subunit is used to train the table recognition model according to the validity classification loss, the coordinate regression loss and the connectivity classification loss.

31. According to the device of any one of claims 24-30, the training device of the table recognition model further comprises a table area detection module, the table area detection module include: A cascade detection unit, used to sequentially perform table area detection on a sample image using at least two cascaded detection units in a table area detection model; wherein the intersection-over-union ratio thresholds used by each cascaded detection unit are sequentially increased; The table area detection unit is used to determine the table area detection result of the sample image according to the table area output by the detection unit at the tail as the sample table image.

32. The device according to claim 31, in, The table area detection model at least includes a feature extraction unit, a basic detection unit and at least one auxiliary detection unit; the auxiliary detection unit is correspondingly provided with a pooling unit; The cascade detection unit is specifically used for: Using the feature extraction unit, extracting features from the sample image to obtain a sample feature map; Using the basic detection unit to perform table area detection on the sample feature map to obtain a basic table area; Scaling the feature map corresponding to the base table area using a correspondingly set pooling unit to obtain a scaled feature map; The auxiliary detection unit is used to perform table area detection on the scaled feature map.

33. An electronic device, include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 16.

34. A non-transitory computer-readable storage medium storing computer instructions, in, The computer instructions are used to make a computer execute the method according to any one of claims 1-16.

35. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 16.

Citation Information

Patent Citations

  • Target detection method and device

    CN111488766A

  • Form recognition method and device in image, electronic equipment and storage medium

    CN111814722A

  • Table image recognition method and device

    CN113255566A

  • Table generation method and device, electronic equipment, storage medium and product

    CN113657274A

  • Table structure identification method, system and equipment based on cell detection

    CN114359939A