Table recognition method and device, table semantic recognition method and device

Abstract

The application relates to the technical field of table image processing, in particular to a table recognition method and device and a table semantic recognition method and device, which solve the problem of low table recognition accuracy. The table recognition method comprises the following steps: firstly, recognizing characters in a to-be-recognized table image, determining character positions and character contents of the to-be-recognized table image; then, based on the character positions and the character contents, determining boundaries of at least one cell of the to-be-recognized table image and character contents corresponding to the at least one cell respectively; further, based on the character contents corresponding to the at least one cell respectively, performing a character aggregation operation to determine phrases corresponding to the at least one cell respectively; finally, based on the boundaries of the at least one cell and the phrases corresponding to the at least one cell respectively, performing a merged cell operation on the at least one cell to determine a table structure of the to-be-recognized table image and character contents corresponding to cells in the table after the merged cell operation.

Description

Technical Field

[0001] This application relates to the field of table image processing technology, specifically to a table recognition method and a table recognition device, a table semantic recognition method and a table semantic recognition device, as well as a computer-readable storage medium and an electronic device. Background Technology

[0002] Tables are a common format in documents, capable of representing complex information within multiple cells, thus facilitating quick information retrieval. This is particularly true in the patent field, where many patent specifications present data in tabular form. For example, chemical industry patents use tables to record elemental proportions, while steel industry patents use tables to record the compositional ratios of different steel grades. Researchers frequently need to extract tabular data from these patents during their research. However, much of this tabular data is presented in image form within the specifications, hindering the rapid retrieval of specific tabular data.

[0003] Table recognition methods in related technologies rely heavily on the boundary lines of table images. They have low accuracy in recognizing tables without boundary lines or with incomplete boundary lines, especially for merged cells in table images. Summary of the Invention

[0004] In view of this, embodiments of this application provide a table recognition method and a table recognition device, a table semantic recognition method and a table semantic recognition device, as well as a computer-readable storage medium and an electronic device, which solve the problem of low table recognition accuracy.

[0005] In a first aspect, an embodiment of this application provides a table recognition method, comprising: recognizing characters in a table image to be recognized, and determining the character positions and character content of the table image to be recognized; based on the character positions and character content, determining the boundaries of at least one cell in the table image to be recognized and the character content corresponding to at least one cell; performing a character aggregation operation based on the character content corresponding to at least one cell to determine the phrase corresponding to at least one cell; and performing a cell merging operation on at least one cell based on the boundaries of at least one cell and the phrase corresponding to at least one cell to determine the table structure of the table image to be recognized and the character content corresponding to the cells in the merged table.

[0006] In conjunction with the first aspect of this application, in some embodiments, based on the boundaries of at least one cell and the phrase corresponding to each of the at least one cell, a cell merging operation is performed on at least one cell to determine the table structure of the table image to be identified and the character content corresponding to the cells in the merged table. This includes: inputting the table image to be identified, including the boundaries of at least one cell and the phrase corresponding to each of the at least one cell, into a semantic recognition model to obtain the re-aggregated phrase corresponding to each of the at least one cell, wherein the semantic recognition model is used to re-aggregate adjacent, semantically related phrases to output the re-aggregated phrase corresponding to each of the at least one cell; and performing a cell merging operation based on the boundaries of at least one cell and the re-aggregated phrase corresponding to each of the at least one cell to determine the table structure of the table image to be identified and the character content corresponding to the cells in the merged table.

[0007] In conjunction with the first aspect of this application, in some embodiments, a cell merging operation is performed based on the boundary of at least one cell and the re-aggregated phrase corresponding to each of the at least one cell, including: for each re-aggregated phrase corresponding to the current cell in the re-aggregated phrase corresponding to each of the at least one cell, if the re-aggregated phrase crosses the boundary of at least one cell, a cell merging operation is performed on the cells on both sides of the crossed boundary.

[0008] In conjunction with the first aspect of this application, in some embodiments, a cell merging operation is performed on at least one cell based on the boundaries of at least one cell and the phrase corresponding to at least one cell, to determine the table structure of the table image to be identified and the character content corresponding to the cells in the merged table, including: using a table recognition model, a cell merging operation is performed on at least one cell based on the boundaries of at least one cell and the phrase corresponding to at least one cell, to determine the table structure of the table image to be identified and the character content corresponding to the cells in the merged table, wherein the table recognition model is used to identify the table structure of the table image to be identified and the character content corresponding to the cells in the merged table.

[0009] In conjunction with the first aspect of this application, in some embodiments, the character position includes the horizontal coordinate and the vertical coordinate of the character position. Based on the character position and the character content, determining the boundary of at least one cell of the table image to be recognized and the character content corresponding to at least one cell includes: determining the row boundary of the table image to be recognized based on the vertical coordinate of the character position; determining the column boundary of the table image to be recognized based on the horizontal coordinate of the character position; determining the boundary of at least one cell of the table image to be recognized based on the row boundary and the column boundary; and determining the character content in at least one cell as the character content corresponding to at least one cell.

[0010] In conjunction with the first aspect of this application, in some embodiments, determining the column boundaries of a table image to be recognized based on the horizontal coordinates of character positions includes: generating a heatmap corresponding to the table image to be recognized based on the horizontal coordinates of character positions, wherein the heatmap is used to characterize the positional distribution of characters in the table image to be recognized; and determining the column boundaries of the table image to be recognized based on the heatmap.

[0011] In conjunction with the first aspect of this application, in some embodiments, before determining the column boundaries of the table image to be identified based on the horizontal coordinate of the character position, the method further includes: determining useless characters based on the horizontal coordinate of the character position and column detection conditions, wherein useless characters are characters that do not contribute to column detection, the column detection conditions include a character length threshold, and useless characters include any one of the following characters: title, header; masking useless characters.

[0012] Secondly, one embodiment of this application provides a table semantic recognition method, comprising: determining the table structure of the table image to be recognized and the character content corresponding to the cells in the table based on the table recognition method mentioned in the first aspect; determining the semantic features of the table based on the table structure and the character content corresponding to the cells in the table, wherein the semantic features of the table include at least one of the following features: the type of character content in the table, the type of cells in the table, and the custom type of the table, wherein the custom type of the table is an entity type set based on user requirements.

[0013] In conjunction with a second aspect of this application, in some embodiments, the type of character content in the table includes at least one of the following types: letters, numbers, blanks, and others.

[0014] In conjunction with a second aspect of this application, in some embodiments, the cell type in the table includes at least one of the following types: header, data, and others, wherein the header includes at least one of the following header types: top header, left header, and cross header.

[0015] Thirdly, one embodiment of this application provides a table recognition device, comprising: a character recognition module configured to recognize characters in a table image to be recognized, and determine the character positions and character content of the table image to be recognized; a boundary determination module configured to determine the boundaries of at least one cell in the table image to be recognized and the character content corresponding to at least one cell based on the character positions and character content; an aggregation module configured to perform a character aggregation operation based on the character content corresponding to at least one cell, and determine the phrase corresponding to at least one cell; and a table determination module configured to perform a cell merging operation on at least one cell based on the boundaries of at least one cell and the phrase corresponding to at least one cell, and determine the table structure of the table image to be recognized and the character content corresponding to the cells in the merged table.

[0016] Fourthly, one embodiment of this application provides a table semantic recognition device, comprising: a table recognition module configured to determine the table structure of a table image to be recognized and the character content corresponding to the cells in the table based on the table recognition method mentioned in the first aspect; and a semantic determination module configured to determine the semantic features of the table based on the table structure and the character content corresponding to the cells in the table, wherein the semantic features of the table include at least one of the following features: the type of character content in the table, the type of cells in the table, and a custom type of the table, wherein the custom type of the table is an entity type set based on user requirements.

[0017] Fifthly, one embodiment of this application provides a computer-readable storage medium storing instructions that, when executed by a processor of an electronic device, enable the electronic device to perform the methods mentioned in the first and second aspects above.

[0018] In a sixth aspect, one embodiment of this application provides an electronic device, which includes: a processor; a memory for storing computer-executable instructions; and a processor for executing the computer-executable instructions to implement the methods mentioned in the first and second aspects above.

[0019] This application provides a table recognition method that first identifies characters in a table image to be recognized, determining the character positions and content. Then, based on the character positions and content, it determines the boundaries and corresponding character content of at least one cell in the table image. Next, it performs character aggregation based on the character content of each cell to determine the corresponding phrase for each cell. Finally, based on the boundaries and corresponding phrases of each cell, it merges the cells to determine the table structure and the character content of the cells in the merged table. In other words, this table recognition method determines cell boundaries based on the identified character positions and content, without relying on boundary information in the table image. Therefore, even if the table image has no boundary lines or incomplete boundary lines, the accuracy of table recognition can be guaranteed. This application also determines the corresponding phrases in the table, thereby utilizing the semantic information of the phrases for cell merging, further improving the accuracy of table recognition. Attached Figure Description

[0020] Figure 1 The diagram shown is an application scenario illustration of the table recognition method provided in an embodiment of this application.

[0021] Figure 2 The diagram shown is a flowchart of a table recognition method provided in an embodiment of this application.

[0022] Figure 3 The image shown is a table image to be identified according to an embodiment of this application.

[0023] Figure 4 The image shown is a visualized table image obtained by recognizing characters in a table image to be recognized according to an embodiment of this application.

[0024] Figure 5 The image shown is an embodiment of this application that provides the character position and character content obtained by recognizing characters in a table image to be recognized.

[0025] Figure 6 The table shown is a result of determining the cell boundaries and the corresponding character content of a cell according to an embodiment of this application.

[0026] Figure 7 The table shown is a table following the determination of the phrase corresponding to the cell according to an embodiment of this application.

[0027] Figure 8 The table shown is a result of merging cells according to an embodiment of this application.

[0028] Figure 9The diagram shown is a flowchart of a table recognition method provided in another embodiment of this application.

[0029] Figure 10 The diagram shown is a flowchart of a table recognition method provided in another embodiment of this application.

[0030] Figure 11 The diagram shown is a flowchart of a table recognition method provided in another embodiment of this application.

[0031] Figure 12 The diagram shown is a flowchart of a table recognition method provided in another embodiment of this application.

[0032] Figure 13 The diagram shown is a flowchart of a table recognition method provided in another embodiment of this application.

[0033] Figure 14 The image shows the character position and content after masking useless characters according to an embodiment of this application.

[0034] Figure 15 The diagram shown is a flowchart of a table recognition method provided in another embodiment of this application.

[0035] Figure 16 The image shown is data after standardizing the height of a character, as provided in an embodiment of this application.

[0036] Figure 17 The image shown is a heat map provided in one embodiment of this application.

[0037] Figure 18 The diagram shown is a schematic diagram of the heat map after removing the gaps between the row boundaries, according to an embodiment of this application.

[0038] Figure 19 The diagram shown is a schematic representation of the boundary of aggregated data provided in an embodiment of this application.

[0039] Figure 20 The diagram shown is a flowchart of a table semantic recognition method provided in an embodiment of this application.

[0040] Figure 21 The image shows the semantic features of a custom type of table provided in an embodiment of this application.

[0041] Figure 22 The image shows the semantic features of the character content type provided in one embodiment of this application.

[0042] Figure 23 The image shows the semantic features of a cell type provided in one embodiment of this application.

[0043] Figure 24The diagram shown is a structural schematic of a table recognition device provided in an embodiment of this application.

[0044] Figure 25 The diagram shown is a structural schematic of a table semantic recognition device provided in an embodiment of this application.

[0045] Figure 26 The diagram shown is a structural schematic of an electronic device provided in an embodiment of this application. Detailed Implementation

[0046] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0047] Exemplary scenario

[0048] Figure 1 The diagram shown is an application scenario illustration of the table recognition method provided in an embodiment of this application. Figure 1 The scenario shown includes a server 110 and a user terminal 120 that communicates with the server 110. Specifically, the server 110 uses the table recognition method of this application to determine the table structure of the table image to be recognized and the character content corresponding to the cells in the table, and sends the table structure of the table image to be recognized and the character content corresponding to the cells in the table to the user terminal 120 so that the user can view or use the table structure of the table image to be recognized and the character content corresponding to the cells in the table.

[0049] In some embodiments, server 110 may be a patent search system, and the table image to be identified may be a table image from the patent specification. Client 120 may be a computer, mobile phone, or other device used by R&D personnel, technicians, or other users who need to search for patents.

[0050] Exemplary methods

[0051] Figure 2 The diagram shown is a flowchart illustrating a table recognition method provided in an embodiment of this application. Figure 2 As shown, the table recognition method includes the following steps.

[0052] Step 210: Recognize the characters in the table image to be recognized, and determine the position and content of the characters in the table image to be recognized.

[0053] Specifically, the characters in the table image to be recognized can be Chinese characters, English characters, punctuation marks, or symbols. The character position is the location of the character within the table image. For example, the character position can be the coordinates of the character within the table image. For instance, a set of coordinates can represent the position of the character in the table image; this set of coordinates may include the coordinates of the character's leftmost boundary, rightmost boundary, topmost boundary, and bottommost boundary.

[0054] Figure 3 The image shown is a table image to be identified according to an embodiment of this application. Figure 4 The image shown is a visualized table image obtained by recognizing characters in a table image to be recognized according to an embodiment of this application. Figure 5 The image shown is an embodiment of this application that provides the character positions and content obtained by recognizing characters in a table image to be recognized. For example... Figure 4 As shown, characters in the visualized table image are surrounded by rectangular boxes, indicating that the character has been recognized. Figure 5 It displays the coordinates of the leftmost, rightmost, topmost, and bottommost edges of the character, as well as the character's height and width, the character's content, and the line in which the initially identified character is located.

[0055] Step 220: Based on the character position and character content, determine the boundary of at least one cell of the table image to be recognized and the character content corresponding to at least one cell.

[0056] Specifically, the position between adjacent characters can be defined as a boundary of a cell. For example... Figure 4 As shown, a boundary can be determined between the bottom edge of character 17.3 and the top edge of character 17.8 to form the bottom edge of the cell corresponding to character 17.3 and the top edge of the cell corresponding to character 17.8.

[0057] For example, Figure 6 The table shown is a result of determining the cell boundaries and the corresponding character content of a cell according to an embodiment of this application.

[0058] Step 230: Perform character aggregation operation based on the character content corresponding to at least one cell to determine the phrase corresponding to at least one cell.

[0059] Specifically, aggregation is the operation of concatenating adjacent characters. Character aggregation based on the character content of at least one cell can be: aggregating the character content within a single cell into a phrase.

[0060] For example, Figure 7The following is a table after determining the phrase corresponding to a cell provided by an embodiment of the present application. As Figure 6 and 7 , Figure 6 the characters "preparation" and "(w / v)" in it are aggregated into the phrase "preparation (w / v)". Figure 6 the characters "(22", "°C", and ") under" in it are aggregated into the phrase "(under 22°C)".

[0061] Step 240, based on the respective boundaries of at least one cell and the respective phrases corresponding to at least one cell, perform a cell merging operation on at least one cell to determine the table structure of the table image to be recognized and the character content corresponding to the cells in the table after cell merging.

[0062] Specifically, the cell merging operation is an operation of merging multiple cells into one cell. Since a phrase can embody semantic information, cells corresponding to phrases with semantic relationships can be merged into one cell. Therefore, performing a cell merging operation on at least one cell based on the respective boundaries of at least one cell and the respective phrases corresponding to at least one cell can be to perform a cell merging operation according to the semantic information of the phrase.

[0063] Exemplarily, Figure 8 the following is a table after cell merging provided by an embodiment of the present application. As Figure 8 shown, cells with semantic relationships are merged into one cell.

[0064] Exemplarily, for a blank cell, it can be merged with an adjacent cell into one cell, as Figure 8 shown. For a blank cell, it can also not be merged with an adjacent cell and be a separate cell.

[0065] The table recognition method of the present application determines the boundaries of cells according to the recognized character positions and character contents, and does not rely on the boundary information in the table image to be recognized. Therefore, even if the table image to be recognized has no boundary lines or the boundary lines are incomplete, the accuracy of table recognition can be guaranteed. The present application also determines the phrases corresponding to the table, and thus uses the semantic information of the phrases to perform a cell merging operation, further improving the accuracy of table recognition.

[0066] In addition, the present application can directly recognize the structure and content of the table image to be recognized, without the need for manual preprocessing of the table image to be recognized, achieving automatic end-to-end table recognition and reducing the labor cost.

[0067] Figure 9 The following is a schematic flowchart of the table recognition method provided by another embodiment of the present application. In Figure 2Extending from the illustrated embodiment Figure 9 The illustrated embodiment will be described in detail below. Figure 9 The illustrated embodiments and Figure 2 The differences between the embodiments shown are not repeated here, and the similarities are not repeated here.

[0068] like Figure 9 As shown in the embodiments of this application, the steps of merging at least one cell based on the boundaries of at least one cell and the phrase corresponding to at least one cell, and determining the table structure of the table image to be identified and the character content corresponding to the cells in the merged table, include the following steps.

[0069] Step 910: Input the table image to be identified, which includes the boundary of at least one cell and the phrase corresponding to at least one cell, into the semantic recognition model to obtain the phrase that is re-aggregated for each cell.

[0070] Specifically, the semantic recognition model is used to re-aggregate adjacent phrases with semantic relationships to output at least one re-aggregated phrase corresponding to each cell.

[0071] For example, the semantic recognition model can be a prior model that has the function of aggregating adjacent phrases with semantic relationships.

[0072] Step 920: Based on the boundary of at least one cell and the phrases that are re-aggregated corresponding to each of the at least one cell, perform a cell merging operation to determine the table structure of the table image to be identified and the character content of the cells in the merged table.

[0073] For example, such as Figure 7 As shown, the semantic recognition model can Figure 7 The phrases "physical", "state", and "(at 22℃)" are then combined again into the phrase "physical state (at 22℃)". Next, the cells corresponding to each phrase "physical", "state", and "(at 22℃)" are merged into one cell, as shown in the image. Figure 8 As shown. The phrase "physical condition (at 22℃)" is the text content corresponding to one cell in the merged table.

[0074] By inputting the image of the table to be identified, which includes the boundary of at least one cell and the phrase corresponding to at least one cell, into the semantic recognition model, the re-aggregated phrase corresponding to at least one cell is obtained. The method is simple and efficient, and the accuracy of the aggregated phrases can be further improved by training the semantic recognition model extensively, thereby providing a more accurate data foundation for subsequent cell merging operations.

[0075] Figure 10 The diagram shown is a flowchart illustrating a table recognition method according to another embodiment of this application. Figure 9 Extending from the illustrated embodiment Figure 10 The illustrated embodiment will be described in detail below. Figure 10 The illustrated embodiments and Figure 9 The differences between the embodiments shown are not repeated here, and the similarities are not repeated here.

[0076] like Figure 10 As shown in the embodiments of this application, the steps of merging cells based on the boundaries of at least one cell and the phrases that are re-aggregated corresponding to each of the at least one cell include the following steps.

[0077] Step 1010: For each of the re-aggregated phrases corresponding to the current cell in at least one cell, determine whether the re-aggregated phrase crosses the boundary of at least one cell.

[0078] Step 1020: Merge the cells on both sides of the crossed boundary.

[0079] Specifically, if the phrases that are aggregated again cross the boundary of at least one cell, a cell merge operation is performed on the cells on both sides of the crossed boundary.

[0080] For example, if the phrase "physical state (at 22°C)" is aggregated again and spans two boundaries between three cells, then the cells on either side of each of those two boundaries will be merged.

[0081] Step 1030: Do not merge cells on either side of the crossed boundary.

[0082] Specifically, if the phrases that are aggregated again do not cross the boundary of at least one cell, the cells on both sides of the crossed boundary are not merged.

[0083] In practical applications, the decision to merge cells is made by determining whether the phrases being merged cross the cell boundaries. This method involves minimal computation and offers high efficiency in table recognition.

[0084] Figure 11 The diagram shown is a flowchart illustrating a table recognition method according to another embodiment of this application. Figure 2 Extending from the illustrated embodiment Figure 11 The illustrated embodiment will be described in detail below. Figure 11 The illustrated embodiments and Figure 2 The differences between the embodiments shown are not repeated here, and the similarities are not repeated here.

[0085] like Figure 11 As shown in the embodiments of this application, the steps of merging at least one cell based on the boundaries of at least one cell and the phrase corresponding to at least one cell, and determining the table structure of the table image to be identified and the character content corresponding to the cells in the merged table, include the following steps.

[0086] Step 1110: Using a table recognition model, based on the boundaries of at least one cell and the phrase corresponding to at least one cell, perform a cell merging operation on at least one cell to determine the table structure of the table image to be recognized and the character content corresponding to the cells in the merged table.

[0087] Specifically, the table recognition model is used to identify the table structure of the table image to be recognized and the character content corresponding to the cells in the table after merging cells.

[0088] For example, the table recognition model can be a priori model that has the function of recognizing the table structure of the table image to be recognized and the character content corresponding to the cells in the table after merging cells.

[0089] In practical applications, using a table recognition model to merge cells is a simple and efficient method. Furthermore, the accuracy of merging cell operations can be further improved by extensive training of the table recognition model.

[0090] Figure 12 The diagram shown is a flowchart illustrating a table recognition method according to another embodiment of this application. Figure 2 Extending from the illustrated embodiment Figure 12 The illustrated embodiment will be described in detail below. Figure 12 The illustrated embodiments and Figure 2 The differences between the embodiments shown are not repeated here, and the similarities are not repeated here.

[0091] like Figure 12 As shown in the embodiments of this application, the steps of determining the boundary of at least one cell of the table image to be identified and the character content corresponding to at least one cell based on the character position and character content include the following steps.

[0092] Step 1210: Determine the row boundaries of the table image to be recognized based on the vertical coordinates of the character positions.

[0093] Specifically, the character position includes the horizontal coordinate and the vertical coordinate of the character position.

[0094] For example, such as Figure 5As shown, each character can have leftmost boundary coordinates, rightmost boundary coordinates, topmost boundary coordinates, and bottommost boundary coordinates. The y-coordinate of the character position can be the topmost or bottommost boundary coordinates of the character. Based on the y-coordinate of the character position, determining the row boundaries of the table image to be recognized can be performed as follows: determine the upper boundary of the row containing the character based on the topmost boundary coordinate, and determine the lower boundary of the row containing the character based on the bottommost boundary coordinate.

[0095] Step 1220: Determine the column boundaries of the table image to be recognized based on the horizontal coordinates of the character positions.

[0096] For example, the horizontal coordinate of a character's position can be the coordinates of the character's leftmost and rightmost boundaries. Determining the column boundaries of the table image to be recognized based on the horizontal coordinates of the character's position can be performed as follows: determine the left boundary of the column containing the character based on the leftmost boundary coordinate, and determine the right boundary of the column containing the character based on the rightmost boundary coordinate.

[0097] Step 1230: Determine the boundaries of at least one cell of the table image to be identified based on row and column boundaries.

[0098] For example, such as Figure 6 As shown, after determining the row and column boundaries, the intersection of the row and column boundaries forms the boundary of at least one cell.

[0099] Step 1240: Determine the character content in at least one cell as the character content corresponding to each of the at least one cells.

[0100] Specifically, once the boundaries of at least one cell are determined, the character content within that cell is the character content corresponding to at least one cell.

[0101] By first determining the row and column boundaries, the cell boundaries are formed by the intersection of the row and column boundaries. This method is simple and efficient.

[0102] Figure 13 The diagram shown is a flowchart illustrating a table recognition method according to another embodiment of this application. Figure 12 Extending from the illustrated embodiment Figure 13 The illustrated embodiment will be described in detail below. Figure 13 The illustrated embodiments and Figure 12 The differences between the embodiments shown are not repeated here, and the similarities are not repeated here.

[0103] like Figure 13 As shown in the embodiments of this application, before the step of determining the column boundaries of the table image to be identified based on the horizontal coordinate of the character position, the following steps are also included.

[0104] Step 1310: Identify useless characters based on the horizontal coordinate of the character position and column detection conditions.

[0105] Specifically, useless characters are characters that do not contribute to column detection. Column detection criteria include a character length threshold. For example, useless characters include any of the following: headers and table headings.

[0106] For example, Figure 5 The image shows the character positions and content before blocking useless characters, according to an embodiment of this application. For example... Figure 5 As shown, the lengths of the characters in the first row are 110, 147, and 59. If the column detection condition is set to a character length threshold of 100, the characters in the first row can be identified as useless characters.

[0107] Step 1320: Filter out useless characters.

[0108] For example, Figure 14 The image shows the character position and content after masking useless characters according to an embodiment of this application.

[0109] The accuracy of column detection can be improved by masking characters that do not contribute to column detection.

[0110] Figure 15 The diagram shown is a flowchart illustrating a table recognition method according to another embodiment of this application. Figure 12 Extending from the illustrated embodiment Figure 15 The illustrated embodiment will be described in detail below. Figure 15 The illustrated embodiments and Figure 12 The differences between the embodiments shown are not repeated here, and the similarities are not repeated here.

[0111] like Figure 15 As shown in the embodiments of this application, the step of determining the column boundaries of the table image to be identified based on the horizontal coordinate of the character position includes the following steps.

[0112] Step 1510: Generate a heatmap corresponding to the table image to be recognized based on the horizontal coordinate of the character position.

[0113] Specifically, heatmaps are used to characterize the positional distribution of characters in a table image to be recognized.

[0114] For example, the height of the characters can be standardized before generating the heatmap corresponding to the image of the table to be recognized, thereby uniformly limiting the height of the characters to 1. Figure 16 The image shown is data after standardizing the height of a character, as provided in an embodiment of this application. Figure 17 The image shown is a heat map provided in one embodiment of this application.

[0115] Step 1520: Based on the heatmap, determine the column boundaries of the table image to be identified.

[0116] Specifically, after generating the heatmap corresponding to the table image to be identified, the gaps between the row boundaries in the heatmap can be removed to obtain... Figure 18 The diagram shown. Figure 18 The diagram shown is a schematic representation of a heat map after removing the gaps between row boundaries according to an embodiment of this application. Then, regarding... Figure 18 The diagram shown illustrates the boundaries of the aggregated data, resulting in... Figure 19 The diagram shown is shown in the image. Figure 19 The diagram shown is a schematic representation of the boundary of aggregated data provided in an embodiment of this application.

[0117] Determining column boundaries of a table image to be identified using heatmaps is a simple method with minimal computational cost.

[0118] Figure 20 The diagram shown is a flowchart illustrating a table semantic recognition method provided in an embodiment of this application. Figure 20 As shown in the embodiments of this application, the table semantic recognition method includes the following steps.

[0119] Step 2010: Determine the table structure of the table image to be recognized and the character content corresponding to the cells in the table.

[0120] Specifically, based on the table recognition method of the above embodiments, the table structure of the table image to be recognized and the character content corresponding to the cells in the table are determined.

[0121] Step 2020: Determine the semantic features of the table based on the table structure and the character content corresponding to the cells in the table.

[0122] Specifically, the semantic features of a table include at least one of the following features: the type of character content in the table, the type of cells in the table, and the custom type of the table, wherein the custom type of the table is an entity type set based on user requirements.

[0123] For example, a prior semantic extraction model can be used to extract the semantic features of a table based on its structure and the character content of the cells within the table.

[0124] For example, Figure 21 The image shows the semantic features of a custom type of table provided in an embodiment of this application. For example... Figure 21 As shown, the cells with gray shading represent custom types: units.

[0125] In some embodiments, the type of character content in the table includes at least one of the following types: letters, numbers, blank (NAN), and others.

[0126] Specifically, you can first calculate the confidence level of the preset types to which the character content in the table belongs, and then select the preset type with the highest confidence level as the type of the character content in the table. For example, Figure 21 The character content "Table 3" is analyzed to have a confidence level of 1 / 6 for letters, 5 / 6 for numbers, 0 / 6 for empty, and 0 / 6 for other. Therefore, the type of the character content "Table 3" is determined to be number.

[0127] For example, Figure 22 The image shows the semantic features of character content types provided in an embodiment of this application. For example... Figure 22 As shown, the characters in cells with light gray shading represent numbers, the characters in cells with dark gray shading represent letters, the characters in cells with diagonal shading represent empty cells, and the characters in cells without shading represent other types.

[0128] In some embodiments, the cell type in the table includes at least one of the following types: header, data, and others. The header includes at least one of the following header types: top header, left header, and cross header.

[0129] For example, Figure 23 The image shows the semantic features of a cell type provided in one embodiment of this application. For example... Figure 23 As shown, cells with dark gray shading are of type header, cells with light gray shading are of type data, and cells without shading are of type other.

[0130] Based on the table structure and the character content corresponding to the cells in the table, the semantic features of the table are determined, the table image is digitized, and the attribute features of the table are constructed, which facilitates subsequent operations such as querying data with specific features of the table by users.

[0131] The above text combined Figures 2 to 23 The method embodiments of this application are described in detail below, in conjunction with... Figure 24 and Figure 25 The present application provides a detailed description of the apparatus embodiments. It should be understood that the descriptions of the method embodiments correspond to the descriptions of the apparatus embodiments; therefore, any parts not described in detail can be found in the foregoing method embodiments.

[0132] Exemplary device

[0133] Figure 24 The diagram shown is a structural schematic of a table recognition device provided in an embodiment of this application. Figure 24As shown, the table recognition device 2400 of this application embodiment includes: a character recognition module 2410, a boundary determination module 2420, an aggregation module 2430, and a table determination module 2440.

[0134] Specifically, the character recognition module 2410 is configured to recognize characters in the table image to be recognized, and determine the character positions and content of the table image. The boundary determination module 2420 is configured to determine the boundaries of at least one cell in the table image to be recognized and the corresponding character content of at least one cell based on the character positions and character content. The aggregation module 2430 is configured to perform character aggregation operations based on the corresponding character content of at least one cell, and determine the corresponding phrase of at least one cell. The table determination module 2440 is configured to perform a cell merging operation on at least one cell based on the boundaries of at least one cell and the corresponding phrase of at least one cell, and determine the table structure of the table image to be recognized and the character content of the cells in the merged table.

[0135] In some embodiments, the table determination module 2440 is further configured to input a table image to be identified, including the boundaries of at least one cell and the phrase corresponding to at least one cell, into a semantic recognition model to obtain a re-aggregated phrase corresponding to at least one cell. The semantic recognition model is used to re-aggregate adjacent phrases with semantic relationships to output the re-aggregated phrase corresponding to at least one cell. Based on the boundaries of at least one cell and the re-aggregated phrase corresponding to at least one cell, a cell merging operation is performed to determine the table structure of the table image to be identified and the character content corresponding to the cells in the merged table.

[0136] In some embodiments, the table determination module 2440 is further configured to perform a cell merging operation on the cells on both sides of the crossed boundary if the re-aggregated phrase in each of the re-aggregated phrases corresponding to at least one cell crosses the boundary of at least one cell.

[0137] In some embodiments, the table determination module 2440 is further configured to use a table recognition model to perform a cell merging operation on at least one cell based on the boundaries of at least one cell and the phrase corresponding to at least one cell, thereby determining the table structure of the table image to be recognized and the character content corresponding to the cells in the merged table, wherein the table recognition model is used to recognize the table structure of the table image to be recognized and the character content corresponding to the cells in the merged table.

[0138] In some embodiments, the character position includes the horizontal coordinate and the vertical coordinate of the character position. The boundary determination module 2420 is further configured to: determine the row boundary of the table image to be recognized based on the vertical coordinate of the character position; determine the column boundary of the table image to be recognized based on the horizontal coordinate of the character position; determine the boundary of at least one cell of the table image to be recognized based on the row boundary and the column boundary; and determine the character content in at least one cell as the character content corresponding to each of the at least one cell.

[0139] In some embodiments, the boundary determination module 2420 is further configured to generate a heatmap corresponding to the table image to be recognized based on the horizontal coordinate of the character position, wherein the heatmap is used to characterize the position distribution of characters in the table image to be recognized; and determine the column boundaries of the table image to be recognized based on the heatmap.

[0140] In some embodiments, the table recognition device 2400 further includes a useless character determination module 2450 and a masking module 2460. Specifically, the useless character determination module 2450 is configured to determine useless characters based on the horizontal coordinate of the character position and column detection conditions, wherein useless characters are characters that do not contribute to column detection, the column detection conditions include a character length threshold, and useless characters include any one of the following characters: title, table header. The masking module 2460 is configured to mask useless characters.

[0141] Figure 25 The diagram shown is a structural schematic of a table semantic recognition device provided in an embodiment of this application. Figure 25 As shown, the table semantic recognition device 2500 of this application embodiment includes: a table recognition module 2510 and a semantic determination module 2520.

[0142] Specifically, the table recognition module 2510 is configured to determine the table structure of the table image to be recognized and the character content corresponding to the cells in the table based on the table recognition method of the above embodiment. The semantic determination module 2520 is configured to determine the semantic features of the table based on the table structure and the character content corresponding to the cells in the table, wherein the semantic features of the table include at least one of the following features: the type of character content in the table, the type of cells in the table, and the custom type of the table, wherein the custom type of the table is an entity type set based on user requirements.

[0143] Exemplary electronic devices

[0144] Figure 26 The diagram shown is a structural schematic of an electronic device provided in an embodiment of this application. Figure 26As shown, the electronic device 2600 includes: one or more processors 2601 and a memory 2602; and computer program instructions stored in the memory 2602, which, when executed by the processor 2601, cause the processor 2601 to perform the method as described in any of the above embodiments.

[0145] The processor 2601 may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device to perform desired functions.

[0146] The memory 2602 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 2601 may execute the program instructions to implement the steps in the methods of the various embodiments of this application described above, and / or other desired functions.

[0147] In one example, electronic device 2600 may further include: input device 2603 and output device 2604, these components being connected via a bus system and / or other forms of connection mechanism. Figure 26 Interconnection (not shown in the image).

[0148] In addition, the input device 2603 may also include, for example, a keyboard, a mouse, a microphone, etc.

[0149] The output device 2604 can output various information to the outside. The output device 2604 may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0150] Of course, for the sake of simplicity, Figure 26 Only some of the components of the electronic device 2600 relevant to this application are shown in this illustration, omitting components such as buses, input devices / output interfaces, etc. In addition, the electronic device 2600 may include any other suitable components depending on the specific application.

[0151] Exemplary computer-readable storage media

[0152] In addition to the methods and apparatus described above, embodiments of this application may also be computer program products, including computer program instructions that, when executed by a processor, cause the processor to perform the steps in the methods of any of the above embodiments.

[0153] Computer program products can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this application. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0154] Furthermore, embodiments of this application may also be computer-readable storage media storing computer program instructions thereon, which, when executed by a processor, cause the processor to perform the steps in the methods according to various embodiments of this application described in the "Exemplary Methods" section above.

[0155] Computer-readable storage media may take the form of any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0156] The basic principles of this application have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this application are merely examples and not limitations, and should not be considered as essential features of each embodiment of this application. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the application to the necessity of employing the aforementioned specific details for implementation.

[0157] The block diagrams of devices, apparatuses, devices, and systems involved in this application are merely exemplary embodiments and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0158] It should also be noted that in the apparatus, equipment, and methods of this application, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions of this application.

[0159] The descriptions of the above embodiments disclosed in this application are intended to enable any person skilled in the art to make or use this application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects without departing from the scope of this application. Therefore, this application is not intended to be limited to the aspects shown herein, but rather to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0160] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this application to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

[0161] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications or equivalent substitutions made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A table recognition method, characterized in that, include: The characters in the table image to be identified are identified to determine the position and content of the characters in the table image to be identified; Based on the character position and the character content, determine the boundary of at least one cell of the table image to be recognized and the character content corresponding to each of the at least one cell; Based on the character content corresponding to each of the at least one cell, a character aggregation operation is performed to determine the phrase corresponding to each of the at least one cell; Based on the boundaries of each of the at least one cell and the phrase corresponding to each of the at least one cell, a cell merging operation is performed on the at least one cell to determine the table structure of the table image to be identified and the character content corresponding to the cells in the merged table. The step of merging at least one cell based on its respective boundary and the phrase corresponding to that cell, and determining the table structure of the table image to be recognized and the character content corresponding to the cells in the merged table, includes: The image of the table to be identified, including the boundary of each of the at least one cell and the phrase corresponding to each of the at least one cell, is input into the semantic recognition model to obtain the phrases that are re-aggregated for each of the at least one cell. The semantic recognition model is used to re-aggregate the adjacent phrases that have semantic relationships to output the phrases that are re-aggregated for each of the at least one cell. Based on the boundaries of the at least one cell and the phrases that are re-aggregated corresponding to each of the at least one cell, the cell merging operation is performed to determine the table structure of the table image to be identified and the character content corresponding to the cells in the table after merging.

2. The table recognition method according to claim 1, characterized in that, The cell merging operation, based on the boundaries of the at least one cell and the phrases that are re-aggregated corresponding to each of the at least one cell, includes: For each of the at least one cell's corresponding aggregated phrases, the phrase corresponding to the current cell in the aggregated phrases is... If the phrase that is re-aggregated crosses the boundary of at least one cell, the cell merging operation is performed on the cells on both sides of the crossed boundary.

3. The table recognition method according to claim 1, characterized in that, The step of merging at least one cell based on its respective boundary and the phrase corresponding to that cell, and determining the table structure of the table image to be recognized and the character content corresponding to the cells in the merged table, includes: Using a table recognition model, based on the boundaries of each of the at least one cell and the phrase corresponding to each of the at least one cell, a cell merging operation is performed on the at least one cell to determine the table structure of the table image to be recognized and the character content corresponding to the cells in the merged table. The table recognition model is used to recognize the table structure of the table image to be recognized and the character content corresponding to the cells in the merged table.

4. The table recognition method according to any one of claims 1 to 3, characterized in that, The character position includes the horizontal coordinate and the vertical coordinate of the character position. Determining the boundary of at least one cell of the table image to be recognized and the corresponding character content of each cell based on the character position and the character content includes: Based on the vertical coordinate of the character position, the row boundaries of the table image to be recognized are determined; Based on the horizontal coordinate of the character position, the column boundaries of the table image to be recognized are determined; Based on the row boundaries and the column boundaries, the boundaries of at least one cell of the table image to be identified are determined. The character content in the at least one cell is determined as the character content corresponding to each of the at least one cell.

5. The table recognition method according to claim 4, characterized in that, Determining the column boundaries of the table image to be recognized based on the horizontal coordinate of the character position includes: Based on the horizontal coordinate of the character position, a heatmap corresponding to the table image to be identified is generated, wherein the heatmap is used to characterize the positional distribution of the character in the table image to be identified; Based on the heatmap, the column boundaries of the table image to be identified are determined.

6. The table recognition method according to claim 4, characterized in that, Before determining the column boundaries of the table image to be recognized based on the horizontal coordinates of the character positions, the method further includes: Based on the horizontal coordinate of the character position and the column detection conditions, useless characters are determined. The useless characters are characters that do not contribute to the column detection. The column detection conditions include the length threshold of the character. The useless characters include any one of the following characters: title, table header. The useless characters are masked.

7. A table semantic recognition method, characterized in that, include: Based on the table recognition method according to any one of claims 1 to 6, the table structure of the table image to be recognized and the character content corresponding to the cells in the table are determined; Based on the table structure and the character content corresponding to the cells in the table, the semantic features of the table are determined, wherein the semantic features of the table include at least one of the following features: the type of character content in the table, the type of cells in the table, and the custom type of the table, wherein the custom type of the table is an entity type set based on user requirements.

8. The table semantic recognition method according to claim 7, characterized in that, The character content in the table includes at least one of the following types: letters, numbers, and blank.

9. The table semantic recognition method according to claim 7, characterized in that, The cell types in the table include at least one of the following types: header, data, wherein the header includes at least one of the following header types: top header, left header, cross header.

10. A form recognition device, characterized in that, include: The character recognition module is configured to recognize characters in a table image to be recognized, and to determine the position and content of the characters in the table image to be recognized; The boundary determination module is configured to determine the boundary of at least one cell of the table image to be identified and the character content corresponding to each of the at least one cell based on the character position and the character content. The aggregation module is configured to perform character aggregation operations based on the character content corresponding to each of the at least one cell, and determine the phrase corresponding to each of the at least one cell. The table determination module is configured to perform a cell merging operation on the at least one cell based on the boundaries of each of the at least one cell and the phrase corresponding to each of the at least one cell, thereby determining the table structure of the table image to be identified and the character content corresponding to the cells in the merged table. The step of merging at least one cell based on its respective boundary and the phrase corresponding to that cell, and determining the table structure of the table image to be recognized and the character content corresponding to the cells in the merged table, includes: The image of the table to be identified, including the boundary of each of the at least one cell and the phrase corresponding to each of the at least one cell, is input into the semantic recognition model to obtain the phrases that are re-aggregated for each of the at least one cell. The semantic recognition model is used to re-aggregate the adjacent phrases that have semantic relationships to output the phrases that are re-aggregated for each of the at least one cell. Based on the boundaries of the at least one cell and the phrases that are re-aggregated corresponding to each of the at least one cell, the cell merging operation is performed to determine the table structure of the table image to be identified and the character content corresponding to the cells in the table after merging.

11. A table semantic recognition device, characterized in that, include: The table recognition module is configured to determine the table structure of the table image to be recognized and the character content corresponding to the cells in the table based on the table recognition method according to any one of claims 1 to 6. The semantic determination module is configured to determine the semantic features of the table based on the table structure and the character content corresponding to the cells in the table. The semantic features of the table include at least one of the following features: the type of character content in the table, the type of cells in the table, and the custom type of the table, wherein the custom type of the table is an entity type set based on user requirements.

12. A computer-readable storage medium, characterized in that, The storage medium stores instructions that, when executed by the processor of the electronic device, enable the electronic device to perform the method described in any one of claims 1 to 9.

13. An electronic device, characterized in that, The electronic device includes: processor; Memory used to store executable instructions for a computer; The processor is configured to execute the computer-executable instructions to implement the method described in any one of claims 1 to 9.

Images