A table parsing method and device, equipment and storage medium

Abstract

The application discloses a table analysis method, device and equipment and a storage medium. The method comprises the following steps: receiving image data containing a table; extracting the geometric position features of text blocks, the appearance features of cells and the content features of text information from the image data respectively; separating the visual features on the rows and columns from the appearance features; extracting the semantic features from the geometric position features and the content features as the appearance semantic features and the content semantic features; learning the first cooperative block representing the rows and the columns, the second cooperative block representing the cells and the text boxes and the third cooperative block representing the text boxes and the text information according to the visual features, the appearance semantic features and the content semantic features; fusing the first cooperative block, the second cooperative block and the third cooperative block into a collaborative graph code; and identifying the structural information in the table according to the collaborative graph code. The shallow image representation information is combined with the semantic information in the cells, so that the accuracy of identifying the table is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and in particular to a table parsing method and device, equipment and a storage medium. BACKGROUND

[0002] The archives of the power grid are extensive in origin and diverse in type. A large number of construction archives project files are generated in various production and construction, and most of the archives store information in the form of tables. When the archives are reviewed, consistency review of the division table is required, that is, the file name corresponding to the division table is indexed according to the file directory table, and then the engineering project name marked with a check mark in the division table is checked one by one to see if it exists in the file directory.

[0003] To improve office efficiency, the content in the table is currently automatically analyzed, modified, mined, and visualized by using scripts, so as to determine whether the file name corresponding to the division table is indexed according to the file directory table and whether the engineering project name marked with a check mark in the division table exists in the file directory.

[0004] Currently, the structure of the table is mainly recognized based on deep learning, including a bottom-up method, a top-down method, and an image text generation method. The bottom-up method mainly features detecting table cells and text blocks first, and then classifying the relationship of the cells. The top-down method first divides the rows and columns of the table, and then performs operations such as merging the cells. The image text generation method refers to directly generating a sequence text corresponding to the table structure based on the table image.

[0005] However, the division table in the production and construction archives of the power grid often has a complex table structure and hundreds of cell contents, and problems such as confusion of the merging relationship, error in row and column division, and the like are prone to occur in the analysis of the table structure, which may result in incomplete cell contents, semantic errors, and the like. Missing recognition of a small number of cells may cause large-area errors in the analysis of the table structure, resulting in incomplete recognition of the engineering project names to be reviewed and compared in the division table, and causing omissions. SUMMARY

[0006] The present application provides a table parsing method, device, equipment and storage medium to solve the problem of how to improve the accuracy of analyzing the structure of the table.

[0007] According to an aspect of the present application, a table parsing method is provided, comprising:

[0008] receiving image data containing a table;

[0009] extracting geometric position features of text blocks, appearance features of cells, and content features of text information from the image data, respectively;

[0010] Separate the visual features in rows and columns from the aforementioned appearance features;

[0011] Semantic features are extracted from the geometric location features and the content features, respectively, and used as appearance semantic features and content semantic features.

[0012] Based on the visual features, appearance semantic features, and content semantic features, a first collaborative block representing the relationship between rows and columns, a second collaborative block representing the relationship between cells and text boxes, and a third collaborative block representing the relationship between text boxes and text information are learned.

[0013] The first cooperative block, the second cooperative block, and the third cooperative block are merged into a cooperative graph code;

[0014] The structural information in the table is identified based on the cooperative graph encoding.

[0015] Optionally, the extraction of geometric location features of text blocks, appearance features of cells, and content features of text information from the image data includes:

[0016] Load the first fully connected layer, residual network, second fully connected layer, word vector model, and convolutional layer;

[0017] Optical character recognition is performed on the image data to obtain text blocks, and the text blocks have location information;

[0018] The position information of the text block is input into the first fully connected layer for mapping to obtain geometric position features;

[0019] The image data is input into the residual network to extract image features;

[0020] Perform a region of interest aggregation operation on the image features to obtain features representing the text boxes corresponding to the cells;

[0021] The features of the text box are input into the second fully connected layer for mapping to obtain the appearance features of the cell;

[0022] The text information corresponding to the text box is input into the word vector model for encoding to obtain the text vector;

[0023] The text vector is input into the convolutional layer to perform a convolution operation, thereby obtaining the content features of the text information.

[0024] Optionally, separating the visual features in rows and columns from the appearance features includes:

[0025] Loading separate aggregation modules and long short-term memory networks;

[0026] inputting the appearance features into the separated aggregation module to extract fused features on rows and columns;

[0027] extracting the fused features into first separated features on rows and second separated features on columns;

[0028] inputting the first separated features and the second separated features into the long short-term memory network to fuse into visual features on rows and columns.

[0029] Optionally, the extracting semantic features from the geometric position features and the content features respectively as appearance semantic features and content semantic features comprises:

[0030] loading a self-semantic extractor provided with a multi-head attention mechanism associated by a residual connection;

[0031] constructing the geometric position features into a first directed graph;

[0032] inputting the first directed graph into the self-semantic extractor to extract semantic features as appearance semantic features through the multi-head attention mechanism;

[0033] constructing the content features into a second directed graph;

[0034] inputting the second directed graph into the self-semantic extractor to extract semantic features as content semantic features through the multi-head attention mechanism.

[0035] Optionally, the learning, according to the visual features, appearance semantic features and content semantic features, a first collaborative block representing a relationship between rows and columns, a second collaborative block representing a relationship between a cell and a text box, and a third collaborative block representing a relationship between a text box and text information, comprises:

[0036] loading a cross-context synthesizer provided with a plurality of multi-head attention mechanisms;

[0037] inputting the visual features into the cross-context synthesizer to learn, in parallel through the plurality of multi-head attention mechanisms, the first collaborative block representing the relationship between rows and columns;

[0038] inputting the appearance semantic features and the content semantic features into the cross-context synthesizer to learn, in parallel through the plurality of multi-head attention mechanisms, the second collaborative block representing the relationship between the cell and the text box and the third collaborative block representing the relationship between the text box and the text information.

[0039] Optionally, the identifying structural information in the table according to the collaborative graph encoding comprises:

[0040] loading a structure prediction network having a plurality of prediction blocks;

[0041] The collaborative graph encoding is sequentially input into a plurality of the prediction blocks for processing to identify structural information in the table.

[0042] Optionally, the number of the prediction blocks is three, and each of the prediction blocks has a third fully connected layer, a fourth fully connected layer and an activation layer.

[0043] According to another aspect of the present application, there is provided a table analysis device, comprising:

[0044] An image data receiving module is configured to receive image data containing a table;

[0045] A feature extraction module is configured to extract geometric position features of text blocks, appearance features of cells and content features of text information from the image data respectively;

[0046] A visual feature separation module is configured to separate visual features on rows and columns from the appearance features;

[0047] A semantic feature identification module is configured to extract semantic features from the geometric position features and the content features as appearance semantic features and content semantic features respectively;

[0048] A collaborative block generation module is configured to learn first collaborative blocks representing relationships between rows and columns, second collaborative blocks representing relationships between cells and text boxes and third collaborative blocks representing relationships between text boxes and text information according to the visual features, the appearance semantic features and the content semantic features;

[0049] A collaborative graph encoding generation module is configured to fuse the first collaborative blocks, the second collaborative blocks and the third collaborative blocks into collaborative graph encoding;

[0050] A structural information generation module is configured to identify structural information in the table according to the collaborative graph encoding.

[0051] Optionally, the feature extraction module is further configured to:

[0052] load a first fully connected layer, a residual network, a second fully connected layer, a word vector model and a convolution layer;

[0053] perform optical character recognition on the image data to obtain text blocks, the text blocks having position information;

[0054] input the position information of the text blocks into the first fully connected layer for mapping to obtain geometric position features;

[0055] input the image data into the residual network to extract image features;

[0056] performing a region of interest gathering operation on the image features to obtain features of a text box corresponding to a single cell;

[0057] inputting the features of the text box into the second fully connected layer to obtain appearance features of the single cell;

[0058] inputting text information corresponding to the text box into the word vector model to obtain a text vector;

[0059] inputting the text vector into the convolution layer to perform a convolution operation to obtain content features of the text information.

[0060] Optionally, the visual feature separation module is further configured to:

[0061] loading a separated aggregation module and a long short-term memory network;

[0062] inputting the appearance features into the separated aggregation module to extract fused features on the rows and columns;

[0063] extracting a first separated feature on the rows and a second separated feature on the columns from the fused features;

[0064] inputting the first separated feature and the second separated feature into the long short-term memory network to fuse the first separated feature and the second separated feature into visual features on the rows and the columns.

[0065] Optionally, the semantic feature recognition module is further configured to:

[0066] loading a self-semantic extractor, the self-semantic extractor being provided with a multi-head attention mechanism associated by a residual connection;

[0067] constructing the geometric position features into a first directed graph;

[0068] inputting the first directed graph into the self-semantic extractor to extract features in semantics as appearance semantic features by the multi-head attention mechanism;

[0069] constructing the content features into a second directed graph;

[0070] inputting the second directed graph into the self-semantic extractor to extract features in semantics as content semantic features by the multi-head attention mechanism.

[0071] Optionally, the collaborative block generation module is further configured to:

[0072] loading a cross-context synthesizer, the cross-context synthesizer being provided with a plurality of multi-head attention mechanisms;

[0073] inputting the visual features into the cross-context synthesizer, learning, in parallel through multiple multi-head attention mechanisms, a first collaborative block between rows and columns;

[0074] inputting the appearance semantic features and the content semantic features into the cross-context synthesizer, learning, in parallel through multiple multi-head attention mechanisms, a second collaborative block between cells and text boxes and a third collaborative block between text boxes and text information.

[0075] Optionally, the structure information generation module is further configured to:

[0076] loading a structure prediction network, the structure prediction network having multiple prediction blocks;

[0077] sequentially inputting the collaborative graph encoding into the multiple prediction blocks for processing to identify the structure information in the table.

[0078] Illustratively, the number of the prediction blocks is three, and each of the prediction blocks has a third fully connected layer, a fourth fully connected layer and an activation layer.

[0079] According to another aspect of the present application, there is provided an electronic device, comprising:

[0080] at least one processor; and

[0081] a memory communicatively connected to the at least one processor; wherein

[0082] the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to perform the table parsing method according to any one of the embodiments of the present application.

[0083] According to another aspect of the present application, there is provided a computer readable storage medium storing a computer program, the computer program being configured to enable a processor to implement the table parsing method according to any one of the embodiments of the present application when executed by the processor.

[0084] In the embodiment, the received content includes image data of a table; geometric position features of text blocks, appearance features of cells, and content features of text information are extracted from the image data respectively; visual features on rows and columns are separated from the appearance features; semantic features are extracted from the geometric position features and the content features respectively as appearance semantic features and content semantic features; first collaborative blocks representing the relationship between rows and columns, second collaborative blocks representing the relationship between cells and text boxes, and third collaborative blocks representing the relationship between text boxes and text information are learned according to the visual features, the appearance semantic features, and the content semantic features; the first collaborative blocks, the second collaborative blocks, and the third collaborative blocks are fused into a collaborative graph code; and structural information in the table is recognized according to the collaborative graph code. The embodiment can generate context for each modality of the table, thereby fusing and modulating interactive information between different modalities of the table, and enabling the intra-modality context generation and inter-modality collaboration to be alternately performed in a hierarchical manner through multiple times of stacking context in different dimensions, which enables the intra-modality interaction to be continuously generated from a low layer to a top layer, that is, low-layer context information and high-layer context information of the multi-modality can be cooperated with each other in the entire network, which can learn sufficient shallow image representation information and better combine semantic information of content in the cells, thereby greatly improving the accuracy of recognizing structural information of the table.

[0085] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the present application, nor is it used to limit the scope of the present application. Other features of the present application will become apparent through the following description. BRIEF DESCRIPTION OF DRAWINGS

[0086] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0087] Figure 1 is a flow chart of a table parsing method provided by the first embodiment of the present application;

[0088] Figure 2 is a structural schematic diagram of a neural collaborative graph machine according to the first embodiment of the present application;

[0089] Figure 3 is a structural schematic diagram of a self-semantic extractor and context synthesizer according to the first embodiment of the present application;

[0090] Figure 4 is a structural schematic diagram of a table parsing device according to the second embodiment of the present application;

[0091] Figure 5 is a structural schematic diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0092] In order to make the personnel in the art better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative efforts should fall within the protection scope of the present application.

[0093] It should be noted that the terms "first", "second", and the like in the specification and claims of the present application and the above-described drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or a chronological sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product, or device that includes a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but can include other steps or units that are not clearly listed or inherent to the process, method, product, or device.

[0094] Embodiment One

[0095] Figure 1 A flowchart of a table parsing method according to Embodiment One of the present application is provided, which can be executed by a table parsing device. The table parsing device can be realized in the form of hardware and / or software, and can be configured in an electronic device. As shown in Figure 1 , the method comprises:

[0096] Step 101: receiving image data containing a table.

[0097] In the present embodiment, a neural coordination graph machine (NCGM) is proposed, as shown in Figure 2 , various archives containing tables (such as division tables) in the power grid are converted into image data, and then the image data can be referred to as table image data (TableImage).

[0098] In the table, a plurality of cells are contained, and the cells contain text information. Generally, the plurality of cells are not distributed in a regular manner, and the text information is also not regular.

[0099] The image data is preprocessed such as paging, and is input into a neural collaborative graph machine (NCGM) for processing.

[0100] In step 102, geometric position features of text blocks, appearance features of cells, and content features of text information are extracted from the image data respectively.

[0101] In this embodiment, as shown in FIG. 1, three feature extractions can be performed on the image data, one of which is to extract geometric position features of text blocks from the image data, another of which is to extract appearance features of cells from the image data, and the other of which is to extract content features of text information from the image data. Figure 2 The text block is an area in which text information is aggregated.

[0102] In a specific implementation, a first fully connected layer (FC) of d dimensions, a residual network (ResNet, such as ResNet18), a second fully connected layer of d dimensions, a word vector model (such as wrod2vec), and a convolutional layer (Convolutional layer) can be loaded.

[0103] Optical Character Recognition (OCR) is performed on the image data to obtain text blocks, wherein the text blocks have position information.

[0104] The position information of the text blocks is input into the first fully connected layer for mapping to obtain geometric position features (Geometry).

[0105] The image data is input into the residual network to extract image features.

[0106] A region of interest (ROI) alignment operation is performed on the image features to obtain features representing text boxes corresponding to cells.

[0107] The features of the text boxes are input into the second fully connected layer for mapping to obtain appearance features (Appearance fseg) of the cells.

[0108] The text information corresponding to the text boxes is input into the word vector model (such as wrod2vec) for encoding, and is encoded into a distribution space to obtain text vectors.

[0109] The text vectors are input into the convolutional layer, a convolution operation with a step size of 1 is performed using a convolution kernel of 7x1xd to obtain content features (content) of the text information.

[0110] ​

[0111] Step 103, separate visual features in rows and columns from the appearance features.

[0112] In this embodiment, high-level feature extraction can be performed on the appearance features, and visual features in rows and columns are separated from the appearance features.

[0113] In a specific implementation, as shown in Figure 2 , a split-aggregation module, a long short-term memory network (lstm module) can be loaded.

[0114] The appearance features are input into the split-aggregation module to extract fusion features in rows and columns, and the fusion features are extracted into first separated features in rows and second separated features in columns by using the correlation relationship.

[0115] The first separated features and the second separated features are input into the long short-term memory network for feature enhancement to be fused into visual features in rows and columns.

[0116] Step 104, respectively extract features in semantics from the geometric position features and the content features as appearance semantic features and content semantic features.

[0117] In this embodiment, high-level feature extraction can be performed on the geometric position features and the content features, features in semantics are extracted from the geometric position features, denoted as appearance semantic features, and features in semantics are extracted from the content features, denoted as content semantic features.

[0118] In a specific implementation, as shown in Figure 2 , a self-semantic extractor (ECE (Ego Context Extractor) module) is loaded.

[0119] As shown in Figure 3 , the modality of each feature input into the ECE is constructed as a separate directed graph G ～ ={v, ε} ∈ {G G , G A , G C}, in each decoupled modality of the graph, the corresponding embedding of each text segment bounding box is regarded as a node X={x1, x2,..., x NThe nodes X are connected to each other through edges e v. In the constructed directed graph, each node can be an anchor point or one of the contexts of other nodes. For the graph representation, the CNN (Convolutional Neural Network) has a strong inductive bias and can not be the best choice. To solve this problem, the self-semantic extractor ECE proposed in this embodiment is provided with a multi-head attention mechanism (MHA) associated through a residual connection, which aggregates all three modes of fully connected graph information through the multi-head attention mechanism. The multi-head attention mechanism has few assumptions about the input and can learn to combine local behavior and global information according to the input content, and can better handle the structure parsing in large and complex tables.

[0120] Further, the MHA is provided with a memory compression module, and at this time, the MHA is also called CMHA (Compressed Multi-head Attention):

[0121] MC(H)=Norm(Reshape(x,∈)W h )

[0122] The CMHA can compress the amount of "memory" to the amount of query (Q), so that the CMHA can reduce the number of image pixels, greatly reducing the computational complexity in the multi-head attention mechanism operation.

[0123] In addition, the CMHA in this embodiment is also provided with a residual connection, so that the query (Q) information can be well passed down, which can be defined as:

[0124]

[0125]

[0126] P=MHA(Q,MC(K),MC(V)),

[0127] Where "FFN(■)" is a feedforward layer, and "Add&Norm(■)" represents element addition and layer normalization.

[0128] On the one hand, the geometric position feature can be constructed as a first directed graph, and the first directed graph is input into the self-semantic extractor to extract the semantic feature through the multi-head attention mechanism as the appearance semantic feature.

[0129] On the other hand, the content feature can be constructed as a second directed graph, and the second directed graph is input into the self-semantic extractor to extract the semantic feature through the multi-head attention mechanism as the content semantic feature.

[0130] Step 105, learning a first collaborative block representing the relationship between rows and columns, a second collaborative block representing the relationship between cells and text boxes, and a third collaborative block representing the relationship between text boxes and text information according to visual features, appearance semantic features and content semantic features.

[0131] In the embodiment, the first collaborative block representing the relationship between rows and columns can be learned according to visual features, and the second collaborative block representing the relationship between cells and text boxes and the third collaborative block representing the relationship between text boxes and text information can be learned according to appearance semantic features and content semantic features.

[0132] In a specific implementation, as shown in Figure 2 , a cross-context synthesizer (CCS (Cross Context Synthesizer) Intermodality) can be loaded.

[0133] As shown in Figure 3 , a plurality of multi-head attention mechanisms are arranged in the cross-context synthesizer.

[0134] In each collaborative block (i.e., the first collaborative block, the second collaborative block and the third collaborative block), the extracted feature embedding is constructed as a context graph, which is applied by ECE alone to form a “modal flow”, and the embodiment fuses them together in a collaborative manner and learns the collaborative mode between different modalities. The CCS has three parallel CMHAs, each of which has three modes, and one mode is used as a query while the other two modes are used as keys K and values V, i.e., the query mode explores useful information from the other two modes, so that the CCS selectively fuses individual contextual information of different modalities into modal interactions maintained in the “modal flow”.

[0135] As shown in Figure 2 , the visual features are input into the cross-context synthesizer, and the first collaborative block Ap representing the relationship between rows and columns is learned in parallel through a plurality of multi-head attention mechanisms.

[0136] The appearance semantic features and the content semantic features are input into the cross-context synthesizer, and the second collaborative block Ge representing the relationship between cells and text boxes and the third collaborative block Co representing the relationship between text boxes and text information are learned in parallel through a plurality of multi-head attention mechanisms.

[0137] Step 106, fusing the first collaborative block, the second collaborative block and the third collaborative block into collaborative graph encoding.

[0138] In the embodiment, as shown in Figure 2 , the first collaborative block Ap, the second collaborative block Ge and the third collaborative block Co can be fused into collaborative graph encoding (collaborative graph embeddings) through functions such as concat, which is shown as Based on this, the ith element and the jth element are spliced together along the channel direction to form a vector

[0139] Step 107, identifying the structural information in the table according to the collaborative graph encoding.

[0140] In this embodiment, as shown in Figure 2 the structural prediction (Structure prediction) can be performed according to the collaborative graph encoding to obtain the structural information in the table, for example, the position information of each cell Cell, the row sequence Row and the column number Col of each cell, and the like, to complete the structural analysis of the table.

[0141] In a specific implementation, a structure prediction network can be loaded, and the structure prediction network has a plurality of prediction blocks.

[0142] The prediction block is an encapsulation containing a predicted deep learning structure, and the structures of the prediction blocks can be the same or different, which is not limited in this embodiment.

[0143] Exemplarily, the number of prediction blocks is three, and each prediction block has a third fully connected layer, a fourth fully connected layer and an activation layer (such as a softmax layer), and the processing in each prediction block is to provide full connection operation for the third fully connected layer, to provide full connection operation for the fourth fully connected layer, and to provide activation operation for the activation layer.

[0144] The collaborative graph encoding is sequentially input into the plurality of prediction blocks for processing to identify the structural information in the table.

[0145] In the embodiment, the received content includes image data of a table; geometric position features of text blocks, appearance features of cells, and content features of text information are extracted from the image data respectively; visual features on rows and columns are separated from the appearance features; semantic features are extracted from the geometric position features and the content features respectively as appearance semantic features and content semantic features; first collaborative blocks representing the relationship between rows and columns, second collaborative blocks representing the relationship between cells and text boxes, and third collaborative blocks representing the relationship between text boxes and text information are learned according to the visual features, the appearance semantic features, and the content semantic features; the first collaborative blocks, the second collaborative blocks, and the third collaborative blocks are fused into collaborative graph encoding; and structural information in the table is recognized according to the collaborative graph encoding. The embodiment can generate context for each modality of the table, thereby fusing and modulating interaction information between different modalities of the table, and enabling the intra-modality context generation and inter-modality collaboration to be alternately performed in a hierarchical manner through multiple times of stacking context in different dimensions, which enables the intra-modality interaction to be continuously generated from a low layer to a top layer, that is, low-layer context information and high-layer context information of the multi-modality can be cooperated with each other in the entire network, which can learn sufficient shallow image representation information and better combine semantic information of content in cells, thereby greatly improving the accuracy of recognizing structural information of the table.

[0146] Embodiment Two

[0147] Figure 4 A structural schematic diagram of a table analysis device provided by Embodiment Two of the present application is shown in FIG. 2. As shown in the figure, the device includes: Figure 3

[0148] An image data receiving module 401 is configured to receive image data of a table.

[0149] A feature extraction module 402 is configured to extract geometric position features of text blocks, appearance features of cells, and content features of text information from the image data respectively.

[0150] A visual feature separation module 403 is configured to separate visual features on rows and columns from the appearance features.

[0151] A semantic feature identification module 404 is configured to extract semantic features from the geometric position features and the content features respectively as appearance semantic features and content semantic features.

[0152] A collaborative block generation module 405 is configured to learn first collaborative blocks representing the relationship between rows and columns, second collaborative blocks representing the relationship between cells and text boxes, and third collaborative blocks representing the relationship between text boxes and text information according to the visual features, the appearance semantic features, and the content semantic features.

[0153] ​The cooperative graph coding generation module 406 is configured to fuse the first cooperative block, the second cooperative block and the third cooperative block into cooperative graph coding.

[0154] The structure information generation module 407 is configured to identify structure information in the table according to the cooperative graph coding.

[0155] In an embodiment of the present application, the feature extraction module 402 is further configured to:

[0156] loading a first full connection layer, a residual network, a second full connection layer, a word vector model and a convolution layer;

[0157] performing optical character recognition on the image data to obtain a text block, the text block having position information;

[0158] inputting the position information of the text block into the first full connection layer to map to obtain geometric position features;

[0159] inputting the image data into the residual network to extract image features;

[0160] performing a region of interest aggregation operation on the image features to obtain features representing a text box corresponding to a single cell;

[0161] inputting the features of the text box into the second full connection layer to map to obtain appearance features of the single cell;

[0162] inputting text information corresponding to the text box into the word vector model to encode to obtain a text vector;

[0163] inputting the text vector into the convolution layer to perform a convolution operation to obtain content features of the text information.

[0164] In an embodiment of the present application, the visual feature separation module 403 is further configured to:

[0165] loading a separation aggregation module and a long short-term memory network;

[0166] inputting the appearance features into the separation aggregation module to extract fused features on rows and columns;

[0167] extracting a first separated feature on the rows and a second separated feature on the columns from the fused features;

[0168] inputting the first separated feature and the second separated feature into the long short-term memory network to fuse into visual features on the rows and the columns.

[0169] In an embodiment of the present application, the semantic feature identification module 404 is further configured to:

[0170] Load from the semantic extractor, the semantic extractor is provided with a multi-head attention mechanism associated by residual connection;

[0171] The geometric position feature is constructed as a first directed graph;

[0172] The first directed graph is input into the self semantic extractor, and the semantic feature is extracted by the multi-head attention mechanism as an appearance semantic feature;

[0173] The content feature is constructed as a second directed graph;

[0174] The second directed graph is input into the self semantic extractor, and the semantic feature is extracted by the multi-head attention mechanism as a content semantic feature.

[0175] In an embodiment of the present application, the collaborative block generation module 405 is further used for:

[0176] Load the cross-context synthesizer, the cross-context synthesizer is provided with a plurality of multi-head attention mechanisms;

[0177] The visual feature is input into the cross-context synthesizer, and a plurality of multi-head attention mechanisms are used to learn a first collaborative block representing the relationship between rows and columns in parallel;

[0178] The appearance semantic feature and the content semantic feature are input into the cross-context synthesizer, and a plurality of multi-head attention mechanisms are used to learn a second collaborative block representing the relationship between a cell and a text box and a third collaborative block representing the relationship between a text box and text information in parallel.

[0179] In an embodiment of the present application, the structure information generation module 407 is further used for:

[0180] Load the structure prediction network, the structure prediction network has a plurality of prediction blocks;

[0181] The collaborative graph code is sequentially input into a plurality of prediction blocks for processing to identify the structure information in the table.

[0182] Illustratively, the number of prediction blocks is three, and each prediction block has a third full connection layer, a fourth full connection layer and an activation layer.

[0183] The table analysis device provided by the embodiments of the present application can execute the table analysis method provided by any embodiment of the present application, and has the corresponding function modules and beneficial effects of executing the table analysis method.

[0184] Embodiment three

[0185] Figure 5A structural diagram of an electronic device 10 that can be used to implement embodiments of the present application is shown. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smartphones, wearable devices (e.g., headsets, glasses, watches, etc.), and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the present application described and / or claimed in this document.

[0186] As shown in Figure 5 The electronic device 10 includes at least one processor 11, and a memory, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., communicatively connected to the at least one processor 11, where the memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes according to the computer programs stored in the read-only memory (ROM) 12 or loaded from the storage unit 18 into the random access memory (RAM) 13. Various programs and data required for the operation of the electronic device 10 can also be stored in the RAM 13. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0187] Various components in the electronic device 10 are connected to the I / O interface 15, including an input unit 16, such as a keyboard, a mouse, etc., an output unit 17, such as various types of displays, speakers, etc., a storage unit 18, such as a magnetic disk, an optical disk, etc., and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0188] The processor 11 can be various general and / or special purpose processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 performs various methods and processes described above, such as the parsing method of the table.

[0189] In some embodiments, the parsing method of a table can be implemented as a computer program tangibly embodied in a computer readable storage medium, e.g., storage unit 18. In some embodiments, portions or all of the computer program can be loaded onto the electronic device 10 via, e.g., ROM 12 and / or communication unit 19. When the computer program is loaded onto RAM 13 and executed by processor 11, one or more steps of the parsing method of a table described above can be performed. Alternatively, in other embodiments, processor 11 can be configured to perform the parsing method of a table by any other suitable means, e.g., by means of firmware.

[0190] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a programmable logic device (PLD), a computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0191] Computer programs used to implement the methods of the present application can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the computer program, when executed by the processor of the machine, implements the functions / acts specified in the flowcharts and / or block diagrams. The computer program can be executed entirely on a machine, partially on a machine, partially on a machine as a stand-alone software package, partially on a machine and partially on a remote machine or entirely on a remote machine or server.

[0192] In the context of the present application, a computer-readable storage medium can be a tangible medium that can contain or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. A computer-readable storage medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. More specific examples of a machine-readable storage medium will include one or more lines of a program of instructions in a transitory signal, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0193] To provide for interaction with a user, the systems and techniques described here can be implemented on an electronic device having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the electronic device. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0194] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0195] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through a communication network. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0196] Example 4

[0197] This invention also provides a computer program product, which includes a computer program that, when executed by a processor, implements a table parsing method as provided in any embodiment of this invention.

[0198] In implementing the computer program product, computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0199] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0200] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method of parsing a table, characterized by, The method comprises: receiving image data containing a table of content; extracting geometric position features of text blocks, appearance features of cells, and content features of text information from the image data, respectively; separating visual features on rows and columns from the appearance features; extracting semantic features from the geometric position features and the content features as appearance semantic features and content semantic features, respectively; learning first collaborative blocks representing the relationship between rows and columns, second collaborative blocks representing the relationship between cells and text boxes, and third collaborative blocks representing the relationship between text boxes and text information according to the visual features, the appearance semantic features, and the content semantic features; fusing the first collaborative blocks, the second collaborative blocks, and the third collaborative blocks into a collaborative graph code; recognizing structural information in the table according to the collaborative graph code; wherein the extracting semantic features from the geometric position features and the content features as appearance semantic features and content semantic features, respectively, comprises: loading a self-semantic extractor in which a multi-head attention mechanism associated by a residual connection is arranged; constructing the geometric position features into a first directed graph; inputting the first directed graph into the self-semantic extractor to extract semantic features as appearance semantic features through the multi-head attention mechanism; constructing the content features into a second directed graph; inputting the second directed graph into the self-semantic extractor to extract semantic features as content semantic features through the multi-head attention mechanism; the learning first collaborative blocks representing the relationship between rows and columns, second collaborative blocks representing the relationship between cells and text boxes, and third collaborative blocks representing the relationship between text boxes and text information according to the visual features, the appearance semantic features, and the content semantic features comprises: loading a cross-context synthesizer in which a plurality of multi-head attention mechanisms are arranged; inputting the visual features into the cross-context synthesizer to learn first collaborative blocks representing the relationship between rows and columns through the plurality of multi-head attention mechanisms in parallel; inputting the appearance semantic features and the content semantic features into the cross-context synthesizer to learn second collaborative blocks representing the relationship between cells and text boxes and third collaborative blocks representing the relationship between text boxes and text information through the plurality of multi-head attention mechanisms in parallel.

2. The method of claim 1, wherein, the extracting geometric position features of text blocks, appearance features of cells, and content features of text information from the image data, respectively, comprises: loading a first fully connected layer, a residual network, a second fully connected layer, a word vector model, and a convolutional layer; performing optical character recognition on the image data to obtain text blocks, the text blocks having position information; inputting the position information of the text blocks into the first fully connected layer to map and obtain geometric position features; inputting the image data into the residual network to extract image features; performing region of interest aggregation on the image features to obtain features representing text boxes corresponding to cells; inputting the features of the text boxes into the second fully connected layer to map and obtain appearance features of cells; inputting text information corresponding to the text boxes into the word vector model to encode and obtain text vectors; The text vector is input into the convolution layer to perform a convolution operation to obtain content features of the text information.

3. The method of claim 1, wherein, The separating the visual features on the rows and columns from the appearance features comprises: loading a separated aggregation module and a long short-term memory network; inputting the appearance features into the separated aggregation module to extract fusion features on the rows and columns; extracting first separated features on the rows and second separated features on the columns from the fusion features; inputting the first separated features and the second separated features into the long short-term memory network to fuse into visual features on the rows and columns.

4. The method according to any one of claims 1-3, characterized in that, The identifying the structural information in the table according to the collaborative graph encoding comprises: loading a structure prediction network, the structure prediction network having a plurality of prediction blocks; sequentially inputting the collaborative graph encoding into the plurality of prediction blocks for processing to identify the structural information in the table.

5. The method of claim 4, wherein, The number of the prediction blocks is three, and each of the prediction blocks has a third full connection layer, a fourth full connection layer and an activation layer.

6. A table parsing apparatus characterized by comprising: comprises: an image data receiving module configured to receive image data containing a table; a feature extraction module configured to extract geometric position features of a text block, appearance features of a cell and content features of text information from the image data respectively; a visual feature separation module configured to separate visual features on rows and columns from the appearance features; a semantic feature identification module configured to extract semantic features from the geometric position features and the content features as appearance semantic features and content semantic features respectively; a collaboration block generation module configured to learn a first collaboration block representing a relationship between rows and columns, a second collaboration block representing a relationship between a cell and a text box and a third collaboration block representing a relationship between a text box and text information according to the visual features, the appearance semantic features and the content semantic features; a collaborative graph encoding generation module configured to fuse the first collaboration block, the second collaboration block and the third collaboration block into a collaborative graph encoding; a structural information generation module configured to identify structural information in the table according to the collaborative graph encoding; The semantic feature identification module is further configured to: load a self-semantic extractor, the self-semantic extractor being provided with a multi-head attention mechanism associated by a residual connection; construct the geometric position features into a first directed graph; input the first directed graph into the self-semantic extractor to extract semantic features as appearance semantic features through the multi-head attention mechanism; construct the content features into a second directed graph; input the second directed graph into the self-semantic extractor to extract semantic features as content semantic features through the multi-head attention mechanism; The collaboration block generation module is further configured to: load a cross-context synthesizer, the cross-context synthesizer being provided with a plurality of multi-head attention mechanisms; input the visual features into the cross-context synthesizer to learn the first collaboration block representing the relationship between the rows and the columns through the plurality of multi-head attention mechanisms in parallel; The appearance semantic features and the content semantic features are input into the cross-context synthesizer, and a second collaboration block between a representation cell and a text box and a third collaboration block between a text box and text information are learned in parallel through multiple multi-head attention mechanism learning units.

7. An electronic device, comprising: The electronic device comprises: at least one processor; and a memory connected to the at least one processor in communication; wherein The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the table parsing method of any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is used to enable the processor to implement the table parsing method of any one of claims 1-5 when executed.

Images