A table parsing method and device and electronic equipment

Abstract

The embodiment of the application discloses a table analysis method and device and electronic equipment. The embodiment of the application inputs a target file into a cell detection model to obtain position information of each cell in the target file, then determines a space adjacency matrix for reflecting the row-column relationship between each cell according to the position information of each cell in the target file, simultaneously obtains cell feature information of each cell in the target file, analyzes the cells in the target file according to the cell feature information and the space adjacency matrix to obtain an analysis result, and finally obtains the corresponding relationship between each two cells in the table to determine the effective information in the table.

Description

Technical Field

[0001] This application relates to the field of deep learning, and in particular to a table parsing method, apparatus, and electronic device. Background Technology

[0002] Tables are a convenient way to represent information in a structured format, suitable for establishing and presenting relational data, and are one of the most important data objects in a document. With the rapid development of deep learning technology, technologies such as object detection, OCR (Optical Character Recognition) engines, and document structure recognition have also made significant progress, providing various table recognition solutions. Currently, there are many solutions in the field of table recognition. However, after obtaining the table structure, there is still no universally applicable and effective method for further parsing the data in the table to extract useful information, such as obtaining the correspondence between data points. Summary of the Invention

[0003] This application discloses a table parsing method and apparatus to determine the cells that make up the table, and further parse each cell in the table to obtain the valid information in the table.

[0004] According to a first aspect of the embodiments of this application, a table parsing method is provided, the method being applied to an electronic device, including:

[0005] The target file is input into the cell detection model to obtain the position information of each cell in the target file;

[0006] Based on the position information of each cell in the target file, a spatial adjacency matrix is ​​determined to reflect the row and column relationships between the cells.

[0007] Obtain the cell feature information of each cell in the target file; the cell feature information is used to identify the cell.

[0008] The cells in the target file are parsed based on the cell feature information and the spatial adjacency matrix to obtain the parsing result. The parsing result includes at least: the key-value pair relationship between any two different cells; the key-value pair relationship between the two different cells is used to indicate that the contents represented by the two cells have a corresponding relationship, and the probability that the corresponding relationship is true.

[0009] Optionally, the cell detection model includes at least: a table detection layer, a local mask alignment layer, a global mask alignment layer, and a post-processing layer.

[0010] The table detection layer is used to detect table regions in the target file;

[0011] The local mask alignment layer is configured to perform non-empty cell detection on the table region to obtain at least one candidate region of a non-empty cell, and obtain a local mask of each candidate region.

[0012] The global mask alignment layer is configured to determine a position of a table frame in the table region and a global mask of the table region.

[0013] The post-processing layer is configured to, for each candidate region, perform optimization on the candidate region according to the local mask of the candidate region and a mask corresponding to the candidate region in the global mask, and take coordinates of the optimized candidate region as coordinates of a non-empty cell corresponding to the candidate region; and determine coordinates of an empty cell according to the position of the table frame and the coordinates of the non-empty cell.

[0014] Optionally, the local mask alignment layer is configured to perform non-empty cell detection on the table region to obtain at least one candidate region of a non-empty cell, and obtain a local mask of each candidate region, including:

[0015] determining, in the table region, a cell border of each non-empty cell and a position of a text block in the non-empty cell, and taking an area in each cell border as a candidate region corresponding to the non-empty cell;

[0016] predicting a horizontal soft label value corresponding to each pixel point in at least one row of pixel points in each cell border; the horizontal soft label value is used to indicate a confidence degree of the cell;

[0017] predicting a vertical soft label value corresponding to each pixel point in at least one column of pixel points in each cell border; the vertical soft label value is used to indicate a confidence degree of the cell;

[0018] for each cell border, taking the horizontal soft label value and the vertical soft label value corresponding to each pixel point as a local mask of a candidate region corresponding to the cell border.

[0019] Optionally, obtaining cell feature information of each cell in the target file includes:

[0020] for each cell in the target file, determining a position feature corresponding to position information of the cell in the target file according to the position information;

[0021] performing semantic analysis on the cell to obtain a semantic feature of the cell; the semantic feature of the cell is used to indicate data content in the cell;

[0022] performing image feature extraction on an image of the cell in the target file to obtain an image feature of the cell;

[0023] Determine at least one of the position feature, the semantic feature and the image feature of the cell as the cell feature information of the cell.

[0024] Optionally, the semantic analysis of the cell to obtain the semantic feature of the cell comprises:

[0025] Input the position information of the cell and the target file into a text recognition tool to obtain the data content in the cell;

[0026] Input the data content in the cell into a semantic feature extraction model to obtain the semantic feature of the cell.

[0027] Optionally, the image feature extraction of the image of the cell in the target file comprises:

[0028] Input the position information of the cell and the cell feature image into an image feature extraction model to obtain the image feature of the cell;

[0029] The cell feature image is used to indicate the position of the cell in the target file.

[0030] Optionally, the analysis of the cells in the target file according to the cell feature information and the spatial adjacency matrix to obtain an analysis result comprises:

[0031] Input the cell feature information and the spatial adjacency matrix into a graph convolutional neural network to obtain the key-value pair relationship between any two different cells in the target file;

[0032] The key-value pair relationship between the two different cells is determined by at least one specified operation on the cell feature information of the two different cells, respectively.

[0033] According to a second aspect of the embodiments of the present application, a table analysis device is provided, which is applied to an electronic device and comprises:

[0034] A position information acquisition unit is configured to input a target file into a cell detection model to obtain the position information of each cell in the target file.

[0035] An adjacency matrix acquisition unit is configured to determine a spatial adjacency matrix reflecting the row-column relationship between each cell according to the position information of each cell in the target file.

[0036] A feature information acquisition unit is configured to obtain the cell feature information of each cell in the target file; the cell feature information of the cell is used to identify the cell.

[0037] The table analysis unit is configured to analyze the cells in the target file according to the cell feature information and the spatial adjacency matrix to obtain an analysis result, the analysis result at least including: a key-value pair relationship between any two different cells; the key-value pair relationship between the two different cells being used to indicate that the contents represented by the two cells have a corresponding relationship and a probability at which the corresponding relationship is established.

[0038] Optionally, the cell detection model in the position information acquisition unit at least includes: a table detection layer, a local mask alignment layer, a global mask alignment layer, and a post-processing layer.

[0039] The table detection layer is configured to detect a table region in the target file.

[0040] The local mask alignment layer is configured to perform text detection on the table region to obtain at least one text block, and determine candidate position information of non-empty cells from the table region according to the text block.

[0041] The global mask alignment layer is configured to determine a position of a table frame line in the table region, perform text detection on the table region to obtain at least one text block, and determine candidate position information of each cell from the table region according to the table frame line and the text block.

[0042] The post-processing layer is configured to process the candidate position information of the non-empty cells determined by the local mask alignment layer according to the candidate position information of each cell determined by the global mask alignment layer, to obtain position information of the non-empty cells and position information of empty cells.

[0043] According to a third aspect of the embodiments of the present application, an electronic device is provided, which includes: a processor and a memory.

[0044] The memory is configured to store machine executable instructions.

[0045] The processor is configured to read and execute the machine executable instructions stored in the memory, to implement the table analysis method as described above.

[0046] The technical solutions provided by the embodiments of the present application can have the following beneficial effects:

[0047] According to the technical solution, the position information of each cell in the target file is obtained by inputting the target file into the cell detection model, and then the spatial adjacency matrix reflecting the row-column relationship between each cell is determined according to the position information of each cell in the target file, and the cell feature information of each cell in the target file is obtained, the cell feature information and the spatial adjacency matrix are used to analyze the cells in the target file to obtain an analysis result, so as to obtain the corresponding relationship between each two cells in the table, and determine the effective information in the table.

[0048] It should be understood that the foregoing general description and the following detailed description are only exemplary and explanatory, and are not limiting to the present application. BRIEF DESCRIPTION OF DRAWINGS

[0049] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present specification and serve to explain the principles of the present specification, together with the description.

[0050] Figure 1 A flowchart of a table analysis method provided by an embodiment of the present application;

[0051] Figure 2 Schematic diagrams of two table structures provided by an embodiment of the present application;

[0052] Figure 3 A schematic diagram of a target file provided by an embodiment of the present application;

[0053] Figure 4 A schematic diagram of horizontal soft label values and vertical soft label values of pixel points in a cell border provided by an embodiment of the present application;

[0054] Figure 5 A schematic diagram of a table analysis device provided by an embodiment of the present application;

[0055] Figure 6 A schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0056] The exemplary embodiments will be described in detail herein below with reference to the drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of apparatuses and methods consistent with some aspects of the present application as detailed in the appended claims.

[0057] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0058] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0059] To enable those skilled in the art to better understand the technical solutions provided in the embodiments of this application, and to make the above-mentioned objectives, features and advantages of the embodiments of this application more apparent and understandable, the technical solutions in the embodiments of this application will be further described in detail below with reference to the accompanying drawings.

[0060] See Figure 1 , Figure 1 This is a flowchart illustrating a table parsing method provided in an embodiment of this application. As one embodiment, Figure 1 The process shown can be applied to electronic devices.

[0061] like Figure 1 As shown, the process may include the following steps:

[0062] Step 101: Input the target file into the cell detection model to obtain the position information of each cell in the target file.

[0063] Optionally, in this embodiment, the target file is generally a file containing a table, such as an image, scanned document, PPT document, or WORD document. Alternatively, the target file can be an image file of a table extracted from other files. It should be noted that this embodiment does not limit the format of the table in the target file (the table format refers to whether the cells in the table have borders, such as...). Figure 2The table a in the target file is a framed table, and the table b is an unframed table. In the embodiments of the present application, the structure of the table (i.e., the number of cells in the table and the arrangement of the cells) does not need to be known in advance. It should be noted that not all the cells in the table have data content. In the embodiments of the present application, the cells with data content are referred to as non-empty cells, and the cells without data content are referred to as empty cells.

[0064] In a specific implementation, if the format of the target file is a picture format that meets the requirements of the cell detection model, the target file can be directly input into the cell detection model. If the format of the target file is not a picture format that meets the requirements of the cell detection model, the format of the target file can be first converted into a picture format that meets the requirements of the cell detection model, and then the converted target file is input into the cell detection model. Alternatively, a format conversion layer can be set in the cell detection model, the target file is input into the cell detection model, and the format conversion layer is used to determine that the format of the target file is not a picture format that meets the requirements of the cell detection model, and then the format conversion layer is used to convert the format of the target file into a picture format that meets the requirements of the cell detection model.

[0065] As an embodiment, the cell detection model in the step 101 can include a table detection layer, a local mask alignment layer, a global mask alignment layer, and a post-processing layer. The table detection layer is configured to detect a table region in the target file. The local mask alignment layer is configured to perform non-empty cell detection on the table region to obtain at least one candidate region of a non-empty cell, and obtain a local mask of each candidate region. The global mask alignment layer is configured to determine a position of a table frame in the table region and a global mask of the table region. The post-processing layer is configured to, for each candidate region, perform optimization on the candidate region according to the local mask of the candidate region and a mask corresponding to the candidate region in the global mask, and take the coordinates of the optimized candidate region as the coordinates of a non-empty cell corresponding to the candidate region. The coordinates of an empty cell are determined according to the position of the table frame and the coordinates of the non-empty cell.

[0066] Optionally, the position information of the cell can include at least position coordinates corresponding to the cell, which are determined by a candidate region enclosed by a cell frame and horizontal soft label values and vertical soft label values of each pixel point in the candidate region. The position coordinates corresponding to the cell can be two vertex coordinates on any diagonal line. The two vertex coordinates can be used to determine a rectangular region in the target file, and the rectangular region is the position of the cell in the target file. The specific process of obtaining the horizontal soft label values and the vertical soft label values of the pixel points in the cell frame will be described later. Figure 1The embodiments shown will be described in more detail below, but not described here.

[0067] As an example, as shown in Figure 3 As shown, the target file can be an article containing a table, and the target file contains tables and pure text, and the cell is the basic unit of the table. Therefore, in order to avoid misidentifying the pure text area as a cell when determining the position information of each cell, after the target file is input into the cell detection model, the cell detection model first needs to determine the table area in the target file through the table detection layer.

[0068] Further, after determining the table area, when the cell contains data content, the data content in the cell is generally a text block composed of multiple characters aggregated together, and different cells in the table are divided by a frame line or a blank area, such as Figure 2 As shown, the table with a frame line divides different cells by the frame line, and the table without a frame line divides the cells by a blank area (such as the area indicated by the dashed line in Table b), and the center line of the blank area can be regarded as the boundary line of the cell. Therefore, when detecting the cells in the table, if a text block aggregated together is detected in the table, it can be inferred that the text block is located in a cell, and the cell belongs to a non-empty cell.

[0069] However, it should be noted that the area of the text block is usually different from the area of the cell to which the text block belongs, and the text block does not occupy the entire area of the cell to which it belongs, and the text block is generally a distance away from the border of the cell. For example Figure 2 The text block "Name" in Table a is far away from the right border of the cell, close to the left border of the cell, and close to the upper and lower borders of the cell, but there is also a distance between the text block "Name" and the upper and lower borders of the cell, and again Figure 2 The text block "X" in Table a is located in the center of the cell, and is a distance away from the upper, lower, left and right borders of the cell. Here, if the position coordinates of the non-empty cell are directly determined according to the area of the text block, a large error is likely to occur.

[0070] Therefore, preferably, in order to better make the border of the cell where the determined text block is more accurate, the local mask alignment layer and the global mask alignment layer as described above are introduced in the embodiments of the present application to obtain position information closer to the real area of the cell. It should be noted that the cell detection model introduced above is only an embodiment of the present application, and the present application does not limit the composition of the cell detection model.

[0071] Exemplarily, the present application can construct a local mask alignment layer and a global mask alignment layer in a cell detection model through a neural network structure LGPMA (the LGPMA neural network structure adopts a deep learning network framework Mask RCNN as a basic framework), so as to predict the position information of the cells in the table region after determining the table region. The LGPMA can construct the local mask alignment layer through a Local Pyramid Mask Alignment (LPMA) branch, so as to detect the text blocks in the table region, predict the candidate region of the non-empty cell containing the text block through the text block, obtain the local mask of each candidate region, and construct the global mask alignment layer through a Global Pyramid Mask Alignment (GPMA) branch, so as to predict the frame line in the table, and obtain the global mask of the table region.

[0072] Since the candidate region of the non-empty cell obtained through the local mask alignment layer is not necessarily accurate, for example, the length of the candidate region of the non-empty cell obtained through the local mask alignment layer is actually greater than or less than the actual length of the non-empty cell, after the candidate region of the non-empty cell obtained through the local mask alignment layer, the candidate region needs to be optimized through the local mask corresponding to the candidate region and the mask corresponding to the global mask of the table region, and then the coordinates of the optimized candidate region are taken as the coordinates of the corresponding non-empty cell.

[0073] Further, based on the position of the table frame line in the table region determined through the global mask alignment layer, the positions of all the cells in the table can be known, and after the coordinates of the non-empty cell are determined, the cells with different coordinates from the non-empty cell in the table can be determined as the coordinates of the empty cell.

[0074] Optionally, in specific implementation, the local mask of the candidate region of the non-empty cell obtained based on the local mask alignment layer is actually more accurate than the mask corresponding to the candidate region in the global mask, and the post-processing layer in the embodiment of the present application can not only optimize the candidate region of the non-empty cell, but also correct the position of the table frame line according to the local mask of the candidate region, for example, if the global mask alignment layer misrecognizes one cell as two cells (for example, misrecognizes the cell “Name” in Table b in the above table as two cells aligned with the cell “X” and the cell “X1” respectively), the misrecognized two cells can be merged into one cell according to the local mask of the candidate region, and the frame line between the misrecognized two cells is eliminated. Figure 2

[0075] ​Step 102, according to the position information of each cell in the target file, determine the spatial adjacency matrix for reflecting the row-column relationship between each cell.

[0076] As an embodiment, the row-column relationship between two cells can be determined by judging whether the position information of the two cells has intersection, and the range of the intersection, and then determining the spatial adjacency matrix according to the row-column relationship between the cells.

[0077] For example, the position information of the cells in the above embodiment includes the two vertex coordinates of the cells. When the two vertex coordinates of a cell p are (x1, y1) and (x2, y2), and the two vertex coordinates of another cell q are (x3, y1) and (x4, y2), it is determined that the cell p and the cell q belong to the same row. When the two vertex coordinates of a cell m are (x1, y1) and (x2, y2), and the two vertex coordinates of another cell n are (x1, y3) and (x2, y4), it is determined that the cell m and the cell n belong to the same column.

[0078] If the x2 of the vertex coordinates of the cell m is equal to x1+50, i.e. the vertex coordinates of the cell m are (x1, y1) and (x1+50, y2), and there is a cell k with two vertex coordinates (x1-1, y5) and (x1+48, y6), although the x coordinates of the two cells are not exactly the same, there is a large intersection between the two cells in the x direction (the length of the intersection is 48), and the difference between the length of the intersection and the length of the edge frame of the cell m or the cell k in the x direction is 2, if the difference is less than the preset intersection threshold, it is still determined that the cell m and the cell k belong to the same column. Here, it is considered that although in the actual table, the cells in the same column as the cell m have an intersection in the x direction equal to |x2-x1| (i.e. 50), but the predicted position coordinates of the cells in the embodiment and the real coordinates of the cells may have errors, therefore, an intersection threshold can be set in the embodiment, if the intersection of the two cells in the x direction and the length of the edge frame of any cell in the x direction is less than the intersection threshold, it is still determined that the two cells belong to the same column. Similarly, when determining whether two cells belong to the same row, the intersection threshold can also be used to determine whether the two cells belong to the same row, to allow similar errors to exist.

[0079] It should be noted that if there is a cell with two vertex coordinates (x1, y1) and (x2, y1+50), and another cell with two vertex coordinates (x3, y1) and (x4, y1+24), it is also considered that the two cells belong to the same row (as shown in Figure 2In Table a, the cell containing the data "X" and the cell containing the data "X1" belong to the same row. This situation usually occurs because one cell is a merged cell (the cell border in the y-direction is longer) and the other cell is a regular cell (the cell border in the y-direction is shorter).

[0080] Optionally, the existence of a merged cell and a regular cell is also allowed for error. In this embodiment, when the difference between the intersection of two cells and the length of the shortest border in the y-direction of the two cells is less than the intersection threshold, the two cells are also considered to belong to the same column. For example, if the intersection threshold is set to 5, and the coordinates of the two vertices of one cell are (x1, y1) and (x1+40, y2), and the coordinates of the two vertices of another cell are (x1-2, y3) and (x1+19, y4), the intersection of the two cells is 19, the shortest border in the y-direction of the two cells is 21, and the difference between the two is 3. Since 3 is less than the intersection threshold of 5, the two cells are also considered to belong to the same column.

[0081] The above is just one example of determining the row and column relationship between different cells. This application does not limit how to determine the row and column relationship between different cells.

[0082] In this embodiment of the application, after determining the row and column relationship between different cells, a spatial adjacency matrix can be determined based on the calculation method of the spatial adjacency matrix in the related technology according to the row and column relationship between different cells.

[0083] Step 103: Obtain the cell feature information used to identify each cell in the target file.

[0084] In this embodiment of the application, the cell feature information is used to identify the cell, such as the cell's location in the target file, the cell's image area in the target file, and, if the cell contains characters, the semantic content represented by the characters in the cell.

[0085] Based on the description of the cell feature information of the cell, as an embodiment, the cell feature information can at least include at least one of a position feature, a semantic feature and an image feature of the cell. For each cell in the target file, the position feature corresponding to the position information of the cell in the target file can be determined according to the position information of the cell; the semantic feature of the cell is obtained by performing semantic analysis on the cell, wherein the semantic feature of the cell is used to indicate the data content in the cell; the image feature of the cell is obtained by performing image feature extraction on the image of the cell in the target file. Then at least one of the position feature, the semantic feature and the image feature of the cell is determined as the cell feature information of the cell. In this embodiment, the acquisition manner of the position feature, the semantic feature and the image feature of the cell will be described in detail after the introduction of the flow shown in the figure. Figure 1 The flow shown in the figure is only one execution order of the embodiments of the present application, and the present application is not limited thereto.

[0086] It should be noted that there is no sequence relationship between the above steps 102 and 103, and step 102 can be executed first or step 103 can be executed first. Figure 1 The flow shown in the figure is only one execution order of the embodiments of the present application, and the present application is not limited thereto.

[0087] In step 104, the cells in the target file are analyzed according to the cell feature information and the spatial adjacency matrix to obtain an analysis result; wherein the analysis result at least includes the key-value pair relationship between any two different cells, and the key-value pair relationship between the two different cells is used to indicate that the contents represented by the two cells have a corresponding relationship and the probability of the corresponding relationship being established.

[0088] As an embodiment, the cell feature information and the spatial adjacency matrix can be input into a graph convolutional neural network (GCN) to obtain the key-value pair relationship between any two different cells in the target file. The GCN can determine the key-value pair relationship between the two different cells by performing specified operations on the feature vectors of the two different cells, wherein the specified operations at least include convolution operation and / or feature similarity calculation, and the specific operation manner can refer to related technologies, which will not be described here.

[0089] Optionally, if the cell feature information in the embodiment includes multiple features of the cell, the feature vectors corresponding to the multiple features need to be spliced into one feature vector and input to the GCN. For example, if the cell feature information of a cell in the embodiment includes a position feature and an image feature, the position feature and the image feature of the cell are spliced and input to the GCN as the feature vector of the cell, and the GCN performs specified operation on the spliced vector as one feature vector. The splicing manner is not limited in the application.

[0090] Optionally, the cell feature information of each cell is continuously updated after being input to the GCN, and the updating process is directly affected by the cell feature information of the cells connected thereto, and is not directly affected by the cell feature information of the cells not connected thereto, that is, in the embodiment, each cell in the table is connected to the cells in the same row and the same column, and the updating of the cell feature information of the cell is only affected by these cells.

[0091] Therefore, in the embodiment of the application, after the GCN obtains the row-column relationship between the cells through the spatial adjacency matrix, for any cell, the cell feature information of the cell can be subjected to multiple specified operations according to the cell feature information of the cells adjacent to the cell, and each operation updates the cell feature information of the cell. After the cell feature information of all cells is subjected to multiple specified operations and is updated to obtain the final features of the cells, the GCN can calculate the probability of the corresponding relationship (i.e., the key-value pair relationship) between any two different cells according to the final features of the cells. The closer the probability is to 1, the greater the probability that the two different cells have a corresponding relationship. The corresponding relationship herein refers to the relationship between the cells representing the attributes and the cells representing the attribute contents in the table, such as the corresponding relationship between the cell with the content "height" and the cell with the content "1.70m".

[0092] At this point, the process shown in Figure 1 is completed.

[0093] As can be seen from the process shown in Figure 1 , the application obtains the position information of each cell in the target file by inputting the target file into the cell detection model, then determines the spatial adjacency matrix reflecting the row-column relationship between the cells according to the position information of each cell in the target file, and obtains the cell feature information of each cell in the target file, analyzes the cells in the target file according to the cell feature information and the spatial adjacency matrix to obtain an analysis result, and thus obtains the corresponding relationship between each two cells in the table to determine the valid information in the table.

[0094] The examples above are for ease of understanding only, and the embodiments of this application are not specifically limited.

[0095] The following is combined Figure 4 The following details the specific process for obtaining the horizontal and vertical soft label values ​​of pixels within the cell border in this embodiment:

[0096] As one embodiment, the local mask alignment layer can predict the cell borders of the non-empty cells containing each text block based on the size of each text block in the table area. Then, it uses the area enclosed by each cell border as the candidate area for each non-empty cell. For each pixel within the cell border, it predicts the horizontal soft label value corresponding to each pixel in at least one row of pixels in each cell border, and predicts the vertical soft label value corresponding to each pixel in at least one column of pixels in each cell border. The horizontal and vertical soft label values ​​are used to indicate the confidence level of the cell. In this embodiment, for each cell border, the horizontal and vertical soft label values ​​corresponding to each pixel can be used as a local mask for the candidate area corresponding to that cell border.

[0097] In this embodiment, for each non-empty cell, the local mask alignment layer can output at least three parts: a mask label indicating the position of the text block, wherein the mask of the non-text area in the predicted cell border is 0, and the mask of the text area is 1; the horizontal soft label value corresponding to each pixel in at least one row of pixels in the cell border of the non-empty cell; and the vertical soft label value corresponding to each pixel in at least one column of pixels in the cell border of the non-empty cell.

[0098] Optionally, in this embodiment, for any text block, the horizontal and vertical soft label values ​​of the pixel corresponding to the center of the text block can both be set to 1, and then the horizontal and vertical soft label values ​​of each pixel can be calculated using linear interpolation. For example... Figure 4 As shown in Figure a, the closer the horizontal distance to the pixel corresponding to the center of the text block, the larger the vertical soft label value of that pixel will be. Figure 4 As shown in Figure b, the closer the vertical distance is to the pixel corresponding to the center of the text block, the larger the horizontal soft label value of that pixel will be.

[0099] As an example, when the global mask alignment layer determines the global mask of the table region, it predicts the horizontal and vertical soft label values ​​of each pixel in the region corresponding to the non-empty cells in each cell determined by the table frame. It sets the mask corresponding to the table frame to 0 and uses the mask of the region corresponding to the non-empty cells and the mask of the table frame as the global mask of the table region.

[0100] In the embodiment of the present application, the global mask alignment layer can output at least three parts of content: a mask label for indicating the table frame line, wherein the mask of the predicted table frame line is 0, and the mask of other areas in the table is 1; the horizontal soft label value corresponding to each pixel point in at least one row of pixel points in the border of each non-empty cell in the entire table structure constituted by the table frame line; and the vertical soft label value corresponding to each pixel point in at least one column of pixel points in the border of each non-empty cell in the entire table structure constituted by the table frame line.

[0101] It should be noted that if the horizontal soft label value or the vertical soft label value of the pixel point on the frame line of the cell border of the non-empty cell is not 0, it indicates that the frame line of the current predicted cell border is inaccurate, and the frame line of the cell border can be further predicted according to the horizontal soft label value or the vertical soft label value of the pixel point on the frame line.

[0102] At this point, the position information of each cell is obtained through the local mask alignment layer, the global mask alignment layer and the post-processing layer in the cell detection model.

[0103] Optionally, the structure of the table in the target file (i.e., the row number and the column number of each cell in the table) can also be determined according to the spatial adjacency matrix, and the cells are identified by the row number and the column number:

[0104] In the embodiment, the cells in the same row can be divided into a group, or the cells in the same column can be divided into a group. Since a cell can belong to two rows of cells in the same row, for example, the cell with data content "X" in Figure 2 belongs to the same row as the cells with data content "X1", "1", "0", respectively, and belongs to the same row as the cells with data content "X2", "0", "1", respectively, the cell with data content "X" will belong to multiple groups: the cell with data content "X" will be divided into a group with the cells with data content "X1", "1", "0", respectively, and will be divided into a group with the cells with data content "X2", "0", "1", respectively.

[0105] Then, the vertex coordinates of only the cells belonging to each group (i.e., indicating that the cell is a normal cell) in each group of cells can be taken as the two vertex coordinates corresponding to the group of cells, the average y coordinate of each group of cells can be calculated by the y coordinates of the two vertex coordinates corresponding to the group of cells, the cell in the group with the smallest value of the average y coordinate can be taken as the first row cell, the row with the second smallest value of the average y coordinate can be taken as the second row, and so on, to determine the row corresponding to each group of cells. In the embodiment, the row number of each cell is the row number of the group to which it belongs, for example, Figure 2If the "X" belongs to two groups, its row number is 2, 3, indicating that it is a merged cell, in the second and third rows in the table.

[0106] After determining the row number of each cell, the column number of each cell can be determined in the same way. The top coordinates of only the cells belonging to the group in each group of cells (i.e., indicating that the cell is a normal cell) can be taken as the two top coordinates corresponding to the group of cells. The average x coordinate of the group of cells is calculated by the x coordinates of the two top coordinates corresponding to the group of cells. The column with the smallest average x coordinate value is taken as the first column, the column with the second smallest average x coordinate value is taken as the second column, and so on, to determine the columns corresponding to each group of cells.

[0107] The specific acquisition method of the cell feature information (at least including the position feature, semantic feature and image feature of the cell) is described below:

[0108] Position feature: As an embodiment, after obtaining the position information corresponding to each cell, the position coordinates in the position information corresponding to each cell can be converted into a feature vector, and the feature vector is taken as the position feature corresponding to the cell. For example, the Positional Encoding (Positional Encoding) technology can be used to convert the two position coordinates corresponding to a cell into a feature vector.

[0109] Semantic feature: As an embodiment, it needs to be pointed out that in this embodiment, only the non-empty cells with text blocks have corresponding semantic features. Semantic analysis of the non-empty cells can obtain the semantic features of the non-empty cells. In specific implementation, the target file and the position information of the non-empty cells can be input into a text recognition tool, so that the text recognition tool performs text recognition on the text blocks in the non-empty cells according to the position information of the non-empty cells, obtains the characters in the text blocks, and then inputs the obtained characters into a pre-trained semantic feature extraction model to convert the characters into vector format semantic features.

[0110] For example, the above-mentioned text recognition tool can be an OCR engine or a PDF parser. When using the OCR engine, the format of the target file needs to be converted into the picture format required by the OCR engine before inputting the target file into the OCR engine. When using the PDF parser, it is necessary to ensure that the target file is a parseable PDF file, such as a PDF file generated by using Word conversion or a typesetting system LaTex.

[0111] Further, in this embodiment, the OCR engine can be used in two ways to perform text recognition on the text blocks in the non-empty cells:

[0112] In a first mode, the position information of the cells obtained in step 101 is first acquired, and the characters in the cells are recognized and recorded according to the position information. The position information of the cells at least includes the position coordinates of the non-empty cells and / or the position coordinates of the text blocks in the non-empty cells.

[0113] In a second mode, the position of the table region in the target file is first acquired, the text recognition is performed on each text block in the table region at a line level to obtain the position coordinates of each line of text and the characters recognized in each line of text, and then the correspondence between the characters recognized in each line of text and the cells is determined according to the position information of the cells obtained in step 101 and the position coordinates of each line of text.

[0114] In the embodiments of the present application, when the characters in the non-empty cells are recognized, the correspondence between the cells and the characters can also be identified in combination with the row numbers and column numbers of the cells obtained above.

[0115] Optionally, after the characters in the non-empty cells are recognized, the characters in the non-empty cells can be input into a semantic feature extraction model to perform semantic feature extraction, to obtain the semantic features in vector format corresponding to the non-empty cells. The semantic feature extraction model can use a semantic feature extraction model trained in related technologies. In specific implementation, if a cell contains multiple lines of characters, the semantic feature extraction can be performed by specifying a symbol (such as a [cls] symbol) to identify that the multiple lines of characters recognized currently belong to the same cell, or the lines of characters in the cell can not be identified, and the characters in the cell are uniformly treated as one line.

[0116] Image feature: as an embodiment, the image feature of a cell is used to indicate the layout information of the cell, and the layout information of the cell refers to the size of the cell, the font format (such as bold, italic, font in Song or Kai) of the characters in the cell, the format (such as text centering or text left or right alignment) of the text lines in the cell, etc.

[0117] Optionally, the image feature of each cell can be obtained by inputting the position information of the cell and a cell feature image into an image feature extraction model, and the image feature of the cell is obtained by the image feature extraction model, and the format of the image feature is also a vector format. The cell feature image can be an intermediate feature calculated by the cell detection model when determining the position information of the cell.

[0118] It should be noted that the cell detection model usually needs to use convolution operation multiple times for feature extraction when extracting the position information of a cell from the target file. In the process of feature extraction, the pixel size of the feature map will be scaled. For example, the original target file has a pixel size of 256*256, and after several convolutions, a feature map with a pixel size of 128*128 is obtained, and then the pixel size of the feature map will be 64*64, 32*32, and so on. Different pixel sizes of the feature map contain different layout information. Generally speaking, the larger the size of the feature map, the more local information it contains, and the smaller the size of the feature map, the more global information it contains. According to the position of the cell in the target file and the scaling ratio of the pixel size of the feature map and the target file, the corresponding position of the cell in each size of the feature map can be calculated.

[0119] In a specific implementation, since the cell detection model scales the pixels of the target file multiple times, in the embodiment, the feature map of a specified scaling ratio can be obtained according to a pre-set instruction, and the feature map and the position information of the cell in the target file are input into the image feature extraction model to obtain the image feature of the cell.

[0120] Further, when the image feature extraction model determines the image feature of the cell, the corresponding position area of the cell in each cell feature image can be calculated according to the position information of the cell, and then the position area image is extracted from each cell feature image, and the multiple position area images extracted from each cell feature image are spliced to obtain the image feature of the cell.

[0121] In the embodiment of the present application, if the cell feature includes at least two of the position feature, the semantic feature and the image feature, after obtaining the corresponding position feature, semantic feature and image feature of each cell through the above steps, at least two of the position feature, the semantic feature and the image feature of each cell can be spliced using a specified algorithm. The specified algorithm for splicing features can refer to related technologies.

[0122] The description of the method embodiment provided by the embodiment of the present application is completed. The device embodiment provided by the embodiment of the present application is described below.

[0123] Referring to Figure 5 , Figure 5 A schematic diagram of a table analysis device provided by the embodiment of the present application is shown. As an embodiment, Figure 5 The device shown in the figure can be applied to an electronic device.

[0124] As shown in Figure 5 , the device includes:

[0125] The position information acquisition unit 601 is configured to input a target file into a cell detection model to obtain position information of each cell in the target file.

[0126] The adjacency matrix acquisition unit 602 is configured to determine a spatial adjacency matrix reflecting the row-column relationship between each cell according to the position information of each cell in the target file.

[0127] The feature information acquisition unit 603 is configured to obtain cell feature information of each cell in the target file; the cell feature information of the cell is used to identify the cell.

[0128] The table analysis unit 604 is configured to analyze the cells in the target file according to the cell feature information and the spatial adjacency matrix to obtain an analysis result, the analysis result at least including a key-value pair relationship between any two different cells; the key-value pair relationship between the two different cells is used to indicate that the contents represented by the two cells have a corresponding relationship and a probability at which the corresponding relationship is established.

[0129] Optionally, the cell detection model in the position information acquisition unit 601 at least includes a table detection layer, a local mask alignment layer, a global mask alignment layer and a post-processing layer.

[0130] The table detection layer is configured to detect a table region in the target file.

[0131] The local mask alignment layer is configured to perform non-empty cell detection on the table region to obtain at least one candidate region of a non-empty cell, and obtain a local mask of each candidate region.

[0132] The global mask alignment layer is configured to determine a position of a table frame line in the table region and a global mask of the table region.

[0133] The post-processing layer is configured to, for each candidate region, perform optimization on the candidate region according to the local mask of the candidate region and a mask corresponding to the candidate region in the global mask, and take the coordinates of the optimized candidate region as the coordinates of a non-empty cell corresponding to the candidate region; and determine the coordinates of an empty cell according to the position of the table frame line and the coordinates of the non-empty cell.

[0134] Optionally, the local mask alignment layer performs non-empty cell detection on the table region to obtain at least one candidate region of a non-empty cell, and obtain a local mask of each candidate region includes:

[0135] determining a cell frame of each non-empty cell and a position of a text block in the non-empty cell in the table region, and taking a region surrounded by the cell frame as a candidate region corresponding to the non-empty cell.

[0136] predicting a horizontal soft label value corresponding to each pixel point in at least one row of pixels in each cell border; the horizontal soft label value is used to indicate the confidence of the cell;

[0137] predicting a vertical soft label value corresponding to each pixel point in at least one column of pixels in each cell border; the vertical soft label value is used to indicate the confidence of the cell;

[0138] for each cell border, taking the horizontal soft label value and the vertical soft label value corresponding to each pixel point as a local mask of a candidate region corresponding to the cell border.

[0139] Optionally, the feature information acquisition unit 603 obtains the cell feature information of each cell in the target file, including:

[0140] for each cell in the target file, determining a position feature corresponding to the position information according to the position information of the cell in the target file;

[0141] performing semantic analysis on the cell to obtain a semantic feature of the cell; the semantic feature of the cell is used to indicate the data content in the cell;

[0142] performing image feature extraction on the image of the cell in the target file to obtain an image feature of the cell;

[0143] determining at least one of the position feature, the semantic feature and the image feature of the cell as the cell feature information of the cell.

[0144] Optionally, the feature information acquisition unit 603 performs semantic analysis on the cell to obtain a semantic feature of the cell, including:

[0145] inputting the position information of the cell and the target file into a text recognition tool to obtain the data content in the cell;

[0146] inputting the data content in the cell into a semantic feature extraction model to obtain the semantic feature of the cell.

[0147] Optionally, the feature information acquisition unit 603 performs image feature extraction on the image of the cell in the target file, including:

[0148] inputting the position information of the cell and a cell feature image into an image feature extraction model to obtain the image feature of the cell;

[0149] wherein, the cell feature image is used to indicate the position of the cell in the target file.

[0150] Optionally, the table parsing unit 604 parses the cells in the target file based on the cell feature information and the spatial adjacency matrix to obtain the parsing result, including:

[0151] The cell feature information and the spatial adjacency matrix are input into a graph convolutional neural network to obtain the key-value pair relationship between any two different cells in the target file;

[0152] The key-value pair relationship between the two different cells is determined by performing at least one specified operation on each of the two different cells.

[0153] Correspondingly, embodiments of this application also provide a hardware structure diagram of an electronic device, specifically as follows: Figure 6 As shown, the electronic device can be a device that implements the table parsing method described above. For example... Figure 6 As shown, the hardware architecture includes a processor and memory.

[0154] The memory is used to store machine-executable instructions;

[0155] The processor is configured to read and execute machine-executable instructions stored in the memory to implement the method embodiment of the corresponding table parsing method shown above.

[0156] As one embodiment, the memory can be any electronic, magnetic, optical, or other physical storage device that can contain or store information such as executable instructions, data, etc. For example, the memory can be volatile memory, non-volatile memory, or similar storage media. Specifically, the memory can be RAM (Random Access Memory), flash memory, storage drives (such as hard disk drives), solid-state drives, any type of storage disk (such as optical discs, DVDs, etc.), or similar storage media, or combinations thereof.

[0157] This concludes the process. Figure 6 Description of the electronic device shown.

[0158] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A table parsing method, characterized in that, This method is applied to electronic devices, including: The target file is input into the cell detection model to obtain the position information of each cell in the target file; a cell is the basic unit of a table, and the area of ​​any cell is different from the area of ​​the text block within that cell. Based on the position information of each cell in the target file, a spatial adjacency matrix is ​​determined to reflect the row and column relationship between each cell. The spatial adjacency matrix is ​​determined in the following way: it is determined whether the position information of any two cells in each cell intersects. If so, the row and column relationship between the two cells is determined based on the range of the intersection, and the spatial adjacency matrix is ​​determined based on the row and column relationship between each cell. Obtain the cell feature information of each cell in the target file; the cell feature information is used to identify the cell. The parsing of cells in the target file based on the cell feature information and the spatial adjacency matrix yields a parsing result. This parsing result includes at least: the key-value pair relationship between any two different cells; the key-value pair relationship between the two different cells indicates that the contents represented by the two cells have a corresponding relationship, and the probability that this correspondence is true. The parsing of cells in the target file based on the cell feature information and the spatial adjacency matrix yields the parsing result by: obtaining the row and column relationships between cells using a Graph Convolutional Neural Network (GCN) based on the input spatial adjacency matrix; for each cell, using the GCN, performing multiple specified operations on the cell feature information of the cell based on the cell feature information of the cells adjacent to that cell to obtain the final feature corresponding to that cell; and using the GCN, calculating the key-value pair relationship between any two different cells based on the final features corresponding to each cell.

2. The method according to claim 1, characterized in that, The cell detection model includes at least: a table detection layer, a local mask alignment layer, a global mask alignment layer, and a post-processing layer; The table detection layer is used to detect table regions in the target file; The local mask alignment layer is used to perform non-empty cell detection on the table area to obtain at least one candidate area of ​​non-empty cells, and to obtain a local mask for each candidate area. The global mask alignment layer is used to determine the position of the table borders in the table area, as well as the global mask of the table area; The post-processing layer is used to optimize each candidate region based on its local mask and the mask corresponding to the candidate region in the global mask, and use the coordinates of the optimized candidate region as the coordinates of its corresponding non-empty cell; and to determine the coordinates of the empty cell based on the position of the table border and the coordinates of the non-empty cell.

3. The method according to claim 2, characterized in that, The local mask alignment layer performs non-empty cell detection on the table region to obtain at least one candidate region of non-empty cells, and obtains the local mask for each candidate region as follows: In the table area, determine the cell border of each non-empty cell and the position of the text block inside the non-empty cell, and use the area enclosed by the cell border as the candidate area corresponding to each non-empty cell. Predict the horizontal soft label value corresponding to each pixel in at least one row of pixels in the border of each cell; the horizontal soft label value is used to indicate the confidence level of the cell; Predict the vertical soft label value corresponding to each pixel in at least one column of pixels in the border of each cell; the vertical soft label value is used to indicate the confidence level of the cell; For each cell border, the horizontal and vertical soft label values ​​corresponding to each pixel are used as the local mask of the candidate region corresponding to that cell border.

4. The method according to claim 1, characterized in that, Obtaining the cell feature information of each cell in the target file includes: For each cell in the target file, the position feature corresponding to the position information is determined based on the position information of that cell in the target file; Semantic parsing is performed on the cell to obtain its semantic features; the semantic features of the cell are used to indicate the data content in the cell. Image features of the cell are extracted from the image in the target file. The cell's location features, semantic features, and image features are determined as the cell's cell feature information.

5. The method according to claim 4, characterized in that, The semantic features of the cell obtained by performing semantic parsing on the cell include: The location information of the cell and the target file are input into a text recognition tool to obtain the data content of the cell; The data content in the cell is input into the semantic feature extraction model to obtain the semantic features of the cell.

6. The method according to claim 4, characterized in that, The step of extracting image features from the image of the cell in the target file includes: The cell's location information and cell feature image are input into the image feature extraction model to obtain the cell's image features; The cell feature image is used to indicate the layout information of the cell.

7. The method according to claim 1, characterized in that, The step of parsing cells in the target file based on the cell feature information and the spatial adjacency matrix to obtain the parsing result includes: The cell feature information and the spatial adjacency matrix are input into a graph convolutional neural network to obtain the key-value pair relationship between any two different cells in the target file; The key-value pair relationship between the two different cells is determined by performing at least one specified operation on the cell feature information of the two different cells respectively.

8. A table parsing device, characterized in that, This device is used in electronic devices, including: The location information acquisition unit is used to input the target file into the cell detection model to obtain the location information of each cell in the target file; a cell is the basic unit of a table, and the area of ​​any cell is different from the area of ​​the text block in that cell. The adjacency matrix acquisition unit is used to determine a spatial adjacency matrix reflecting the row and column relationships between cells based on the position information of each cell in the target file. The spatial adjacency matrix is ​​determined by: determining whether the position information of any two cells in each cell intersects; if so, determining the row and column relationship between the two cells based on the range of the intersection, and determining the spatial adjacency matrix based on the row and column relationship between the cells. The feature information acquisition unit is used to obtain the cell feature information of each cell in the target file; the cell feature information is used to identify the cell. The table parsing unit is used to parse the cells in the target file based on the cell feature information and the spatial adjacency matrix to obtain parsing results. The parsing results include at least: key-value pair relationships between any two different cells; the key-value pair relationships between the two different cells are used to indicate whether the contents represented by the two cells have a corresponding relationship and the probability that the corresponding relationship is true; the parsing of the cells in the target file based on the cell feature information and the spatial adjacency matrix to obtain parsing results includes: obtaining the row and column relationships between each cell based on the input spatial adjacency matrix using a graph convolutional neural network (GCN); for each cell, using the GCN, performing multiple specified operations on the cell feature information of the cell based on the cell feature information corresponding to the adjacent cells to obtain the final feature corresponding to the cell; and using the GCN, calculating the key-value pair relationships between any two different cells based on the final features corresponding to each cell.

9. The apparatus according to claim 8, characterized in that, The cell detection model in the location information acquisition unit includes at least: a table detection layer, a local mask alignment layer, a global mask alignment layer, and a post-processing layer; The table detection layer is used to detect table regions in the target file; The local mask alignment layer is used to perform non-empty cell detection on the table area to obtain at least one candidate area of ​​non-empty cells, and to obtain a local mask for each candidate area. The global mask alignment layer is used to determine the position of the table borders in the table area, as well as the global mask of the table area; The post-processing layer is used to optimize each candidate region based on its local mask and the mask corresponding to the candidate region in the global mask, and use the coordinates of the optimized candidate region as the coordinates of its corresponding non-empty cell; and to determine the coordinates of the empty cell based on the position of the table border and the coordinates of the non-empty cell.

10. An electronic device, characterized in that, The electronic device includes: a processor and memory; The memory is used to store machine-executable instructions; The processor is configured to read and execute machine-executable instructions stored in the memory to implement the method according to any one of claims 1 to 7.

Images