A Method for Extracting Table Metric Information Based on Graph Attention Network

Abstract

The present invention belongs to the technical field of digitalization of industry standard information, and specifically relates to a method for extracting tabular index information based on a graph attention network. The present invention includes: table preprocessing; cell information encoding; adjacency list analysis; table information correlation analysis; classification and index information calculation. Table preprocessing is to perform graph network structuring on the table to obtain the graph network structure of the table information; then, word vector encoding and adjacency list analysis are respectively performed on the cells in the table. Adjacency list analysis includes obtaining the row adjacency list and column adjacency list of the cell, and combining with word vector encoding to obtain the semantic information of the cells in the same row or the same column. Table information correlation analysis includes table information correlation analysis and row-column information transmission relationship analysis; classification and index information calculation use a linear classifier to perform classification to obtain the index classification result of the current cell. The present invention is more efficient, and the obtained tabular index information is more accurate and complete.

Description

Technical Field

[0001] The present invention belongs to the technical field of digitalization of industry standard information, and particularly relates to a method for extracting tabular index information based on a graph attention network. Background Art

[0002] With the advancement of digitalization, all industries are carrying out digital construction, including the digitalization of industry standard information. In the past, the extraction of standard indicators was generally carried out by industry experts after interpreting standard documents. Since new standard documents are released or updated every year, manual extraction is time-consuming and inefficient. The introduction of deep learning technology enables information extraction algorithms to extract key information from unformatted text. However, when dealing with tabular information in standard documents, these algorithms cannot fully utilize the structural information and content information of the table part.

[0003] Most of the methods for extracting indicators from standard documents rely on manual annotation by experts. However, there are a large number of specifications in the industry. Using the manual annotation method requires a lot of manpower and time, and it is inevitable to make mistakes during manual annotation. In addition, each expert may have different understandings of the specifications. Therefore, the final annotation results may also vary, and there may even be contradictory situations. For example, for the same indicator name, the division rules given by different experts may be different, the division criteria are not unified, and there are defects such as mislabeling or missing labeling. In addition, due to the large number of standard documents in the rail transit industry, the traditional manual annotation method can no longer meet the actual needs in terms of efficiency and accuracy, and a more efficient method is needed to replace manual annotation for the extraction and management of indicators.

[0004] Traditional information extraction algorithms, such as BiLSTM-CRF, etc., although able to extract key information from large paragraphs of text, usually need to make judgments based on the context of the key information. Therefore, they are applicable to unformatted text. When dealing with tabular information, the difference between tabular information and conventional text information is that the table itself has structured information. Taking standard indicator information as an example, in unformatted text, the key information related to the same standard indicator is usually in the same sentence or two consecutive sentences. However, in tabular information, all the key information involved in a complete indicator information is distributed in multiple tables. If the entire table is directly processed as unformatted text, some related indicators may be far apart in position, making it difficult to obtain information between them. In addition, if the table text is directly processed as unformatted text, although a large number of data cells in the table can be extracted as indicator data, the relationship between these indicator data and other indicator names and indicator categories cannot be determined.

[0005] Currently, there are few relevant studies on directly extracting information from tabular information. The research on information extraction mainly focuses on fields such as large segments of unstructured text, while the information extraction research on other formatted and semi-formatted texts similar to tables mainly focuses on tasks such as Web information extraction and invoice information recognition.

[0006] The graph structure-based method takes the text as a graph structure composed of small pieces, and then uses deep learning methods to learn the relative relationships between text blocks, so as to extract the key information of the document. The traditional named entity recognition NER algorithm BiLSTM-CRF does not make full use of the layout information of the document. When encoding text information, it uses the size and position information of the text box, converts it into position vector information and inputs it into the graph network, and through the graph embedding generated by graph convolution, the current text segment can summarize the context information. Finally, the features learned by GCN are connected to the BiLSTM-CRF module to complete the information classification task. The LayoutLM model proposed by Xu et al. [9] uses BERT as the backbone network, introduces two-dimensional position vectors to represent layout and layout information, uses the horizontal and vertical coordinates of the upper left and lower right corners of each text box to record the position of the entire text box relative to the document, and selectively adds visual vectors to supplement more information, enabling downstream tasks to better combine multi-modal information. Wang et al.

[10] converted the joint task of entity relationship extraction into a directed graph by designing a novel graph algorithm and proposed a transition-based method to gradually generate a directed graph. Riba

[11] and Qasim

[12] et al. respectively did related work on how to convert invoice information into a graph model. Among them, Riba et al. connected all vertically and horizontally aligned paragraphs according to the positions of all recognized text paragraphs, and then converted all text information in the invoice into a graph structure. Qasim et al. used the KNN algorithm to connect each text paragraph as a graph node to its k nearest other nodes. The main feature of the graph structure-based method is that after encoding the position information of each text block and combining it with the text information, it can make full use of the spatial layout information of the document.

[0007] The end-to-end method directly uses the original image information as input to obtain the key information of the document. The main algorithms include EATEN

[13] and TRIE

[14] . Guo et al. found that the information extraction technology based on the detection and recognition process is affected by factors such as slight position offsets, and thus proposed the EATEN algorithm. The EATEN algorithm uses a convolutional neural network to extract the visual information of the image, then uses an attention network based on entity category perception to learn the layout information, and finally uses an LSTM-based decoder to obtain the required entity information. The TRIE algorithm combines the three tasks of text detection, text recognition, and information extraction into an end-to-end network for learning. In the TRIE algorithm, Resnet and FPN are used as the backbone networks to detect the text part, then the text detection network and the results of ROIAlign

[15] are used for text recognition, and finally the text position information, visual information, and semantic information are fused, and a BiLSTM is used for IOB tag classification to obtain the key information of the document.

[0008] In previous studies, the method used to construct the graph structure was usually to establish connections between adjacent nodes in the up, down, left, and right directions, or to connect all the graph nodes on the entire graph. However, this graph construction method is not suitable for table information. First, the relevant information of a cell in table data usually includes other cells in the same row and the same column. If the method of connecting all cells is used, a large amount of interfering data will exist in the connected nodes. If only connected to adjacent cells, a table with a large number of rows and columns requires multiple layers of graph networks to enable cells that are far apart to communicate with each other.

[0009] Since the degree of association between other cells in the same row and the same column and the current cell is different, the impact on their final classification is also different. The previous related research used the standard graph attention network structure, that is, using a shared weight matrix W to calculate and concatenate the information of two graph nodes. However, using only one shared weight matrix W has too few parameters, which affects the model capacity. Therefore, referring to the solution in the Bert algorithm, a table correlation module TCM (Table Correlation Module) is proposed, which uses a query matrix W q and a key-value matrix W k to operate on the text information of the cells, so as to obtain the correlation degree between two cells. After the standard graph attention network calculates the attention of the current graph node to other graph nodes, it calculates the weights according to the attention, aggregates the information of the surrounding nodes, and updates its own information. However, if the attention of a certain other node is greater than the self-attention, the information of the node itself will be replaced by the information of other nodes when updating the information. Summary of the Invention

[0010] The purpose of the present invention is to provide a method for extracting table metrics based on a graph attention network to overcome the deficiencies and drawbacks of the prior art.

[0011] In view of the problem of uncertain table size, the present invention uses a graph network to solve the classification problem of table data, and optimizes it by converting the table structure into a graph structure according to the characteristics of the table structure. The distance between all cells in the same row and the same column is set to one hop, making the graph structure more suitable for processing table data. Based on the improvement of the graph attention network, a table correlation module TCM and a row-column information transmission module TITM are designed, enabling each cell to adaptively obtain the information of other cells, and while retaining the original information of the nodes, simultaneously obtaining the important information of other cells in the entire table, thereby integrating the standard index extraction algorithm for table data. Finally, the effectiveness of the algorithm is verified through experiments, and the index extraction for table data is successfully completed.

[0012] The method for extracting table metric information based on the graph attention network provided by the present invention is divided into five major parts: (1) Table preprocessing; (2) Cell information encoding (i.e., word vector encoding); (3) Adjacency list analysis; (4) Table information correlation analysis; (5) Classification and metric information calculation; see Figure 1 as shown. Among them:

[0013] The so-called table preprocessing is to structurally process the table information into a graph network to obtain the graph network structure of the table information; then, word vector encoding and adjacency list analysis are respectively performed on a certain cell in the table.

[0014] The cell information encoding includes word segmentation and word embedding encoding of the text in the table, and inputting the encoded result into a bidirectional recurrent neural network for semantic feature extraction to obtain the cell semantic information U i ;

[0015] The adjacency list analysis includes obtaining the row adjacency list and column adjacency list of a certain cell, and combining with the word vector encoding to obtain the cell semantic information U j ;

[0016] The table information correlation analysis is to perform table information correlation analysis according to U i and U j ; it includes: table information correlation analysis calculation and row-column information transmission relationship analysis calculation; after two-layer correlation calculation, the semantic information in the row and column directions of the same row or the same column at two layers is obtained. Among them, the row-column semantic information at the first layer is V row(1) and V col(1) , and the row-column semantic information V row(2) and V col(2), Take the correlation information of the first layer as part of the model output; take the row-column semantic information of the second layer as V row(2) and V col(2) and the cell semantic information U i Merge them, and then pass them to the classification output layer for output;

[0017] For classification and index information calculation, use a linear classifier for classification to obtain the index classification result of the current cell.

[0018] The specific content of each step will be further described below:

[0019] (1) Table preprocessing

[0020] In order to utilize the structural information of the table, a structure that can process table information with variable data sizes is designed, that is, the table information is processed by graph network structuring. Specifically, the input table data is regarded as a graph, each cell in the table is regarded as a node in the graph, and adjacent cells are regarded as two graph nodes connected to each other. Thus, the information of the entire table can be transformed into graph information, thereby solving the problem that the sizes of different tables are different. See Figure 2 as shown.

[0021] The graph network obtains information from other surrounding nodes by constructing an adjacency matrix of graph nodes and aggregates this information to continuously update the information contained in its own nodes, but a graph node can only obtain information through its adjacent nodes each time. Different from the convolutional neural network, the convolutional neural network can increase the receptive field by expanding the convolutional kernel, while if the graph network wants to obtain more information of other nodes in the graph, it needs to increase the number of layers and obtain the information of the (i + 1)-th hop neighbor through the i-th hop neighbor.

[0022] However, when the number of layers of the graph network is too large, the multi-hop neighbors of different nodes will be the same, which will cause the information obtained by the graph nodes from other surrounding nodes to gradually converge, resulting in little difference in the information aggregated by each node after passing through multiple layers of the graph network, easily leading to the over-smoothing problem and making the graph network ineffective. In addition, too many layers will also cause the information transmission between the graph network nodes to be too slow. For example, in a large table, for the cells in the first column and the last column, even if there is a certain correlation, the information transmission between them needs to pass through all the intermediate cells at least, which is likely to cause slow information transmission and gradient disappearance. Therefore, it is necessary to avoid too many layers as much as possible when using the graph network.

[0023] A complex table is a table that includes a large number of rows or columns. If each cell and its adjacent neighboring nodes are regarded as one-hop neighbors, when dealing with these tables with a large number of rows or columns, it is inevitable that there will be a problem of excessive layers (the definition method of one-hop neighboring nodes is shown in Figure 3 as shown). If the maximum number of layers of the graph network is directly restricted, it will cause cells to only obtain information of other cells within a certain range, affecting the accuracy of the final cell classification result and the effect of the final indicator extraction. Therefore, the graph structure transformed from the table is adjusted. Instead of taking the four adjacent cells above, below, left, and right of a cell as its one-hop neighbors, all other cells in the same row or the same column of it are taken as its one-hop neighbors, so that each cell can obtain more table content information on the premise of passing through the shortest possible path.

[0024] Using this rule to define the graph structure of the table can enable any two cells that are not in the same row or the same column to also transmit information to each other only after two-hop distance. Therefore, only a two-layer graph network structure needs to be constructed to enable any cell in the table to obtain semantic information within the range of the entire table, thus avoiding the over-smoothing problem and the problem of slow information transmission that may be caused by excessive layers of the graph network.

[0025] In addition, when a certain row or column of the table is exchanged with other rows or columns as a whole, the information contained in the table itself does not necessarily change. Taking Figure 4 as shown in the figure, when the data related to "vehicle fire protection level" in the table is swapped with the data related to "droplet combustion level" in the whole column, the indicator information contained in the whole table does not change. However, if adjacent nodes are used as one-hop neighbors, for the data "ST2" and "2" in the third row, their neighboring nodes change accordingly, and the information obtained also changes. This will lead to different front and back orders of reading the table data, and may also affect the result of the final indicator classification. Therefore, by adopting the above method of redefining neighboring nodes, no matter how the rows and columns change, the neighboring nodes of each cell will not change accordingly, and this scheme is more robust when dealing with table data.

[0026] (2) Cell Information Encoding

[0027] In the word vector encoding layer, the main task is to convert the text information of all tables into a computable vector matrix. This includes tasks such as word segmentation of cells and word vector encoding. For the word segmentation task of cells, the current main word segmentation tools include Jieba, Snownlp, THULAC, etc. After comparing the results of the above word segmentation tools, the Jieba word segmentation tool is selected to segment the text of cells. After word segmentation, the Word2Vec method is used to convert each word into a corresponding word vector and splice them. Since the information of each cell needs to be converted into a fixed size, 15 words are intercepted for each cell, and the length of each word vector is fixed at 300 dimensions. Therefore, after passing through the cell information encoding layer, the text information of the entire table is converted into a word vector matrix of {[number of cells]*15*300}. Subsequently, the information of each cell passes through the bidirectional long short-term memory (BiLSTM) of the recurrent neural network, enabling the words within each cell to obtain the context relationship within the current cell.

[0028] (III) Adjacency List Analysis

[0029] In a graph network, it is usually necessary to rely on an adjacency matrix or an adjacency list to represent the structure of the graph. Since the graph structure directly connects any cell to other cells in the same row or the same column as its one-hop adjacent nodes, the other cells connected to each cell can be calculated based on the row and column positions of each cell. In addition to the text information of the table, the input part of the overall algorithm also needs to input the number of rows row num and the number of columns column num . Subsequently, each cell is numbered in row-major order. Since the information of the number of rows and columns of the table is obtained, each cell D can calculate its row i and col i based on its own index number:

[0030] row i = index div column num , col i = index mod column num , (1)

[0031] By traversing the row coordinates and column coordinates, all adjacent node sets of cell i in the row direction can be obtained and all adjacent node sets in the column direction

[0032]

[0033] Finally, all the calculated adjacent node sets are concatenated to obtain the adjacency list Adj in the row directionrow and the column - direction adjacency list Adj col , for use in the next - step calculation.

[0034] (4) Table information association analysis

[0035] Among them, it includes the calculation of table information association degree analysis and the analysis of the information transfer relationship between rows and columns.

[0036] (1) Calculation of table information association degree analysis

[0037] After determining the graph - network structure representing the table, it is necessary to design the information aggregation method. Currently, each cell can obtain the information of other cells in the same row and the same column, but the information of different cells is different, and the influence on the final classification result of the current cell is also different.

[0038] Take Figure 5 as an example: Through the data value "SR2" at the position (3, 5) in the table, the relevant index classification "luggage rack" can be found in its same row, and the relevant index name "smoke - production level" can be found in its same column. There is a certain semantic - relationship connection between these values. In order to obtain information from its surrounding cells, so as to have context - information supplementation to help judge the category to which it finally belongs. A correlation - degree module is constructed in the graph network to learn the semantic - information correlation degree between two different cells, and according to the information correlation degree, determine the influence degree of other information in the same row and the same column on the current cell, determine the weight of the contribution degree of each cell, and then multiply the obtained weight by the information of other cells in the same row and the same column and add them up, so as to obtain the text information integrating other cells in the whole row or the whole column. The flow - chart of the correlation - degree module is shown in Figure 6 as follows.

[0039] In the original graph - attention network, the attention between two nodes is calculated by the following formula:

[0040]

[0041] where h i and h j represent the feature vectors of the graph nodes, with a size of 1×F, F represents that each graph node has F features, W is a learnable parameter matrix with a size of F×F′, after calculation, a feature vector with a size of 1×F′ is obtained, and then the feature vectors calculated for node i and node j are directly concatenated and mapped to a real number through the function a(·), so as to obtain the attention between node i and node j.

[0042] The information contained in each graph node is the semantic information of several words in the entire cell. That is, the size of the information matrix of each graph node itself is n*d, where n is the number of words in a cell and d is the dimension of the word vector of each word. Therefore, corresponding adjustments need to be made to the network.

[0043] Construct the query matrix W q and the key-value matrix W k , where W q is calculated with the semantic information of the currently selected cell, while W k is calculated with other cells in the same row or the same column as this cell, obtaining the matrices Q and K.

[0044] Q i = W q U i , (3)

[0045]

[0046] W q and W k are both of size d×d. Therefore, the sizes of Q and K are n×d. Subsequently, Q is multiplied by the transposed K to obtain the matrix R:

[0047] R ij = Q i K j T , i = 1, 2,..., n, j = 1, 2,..., n; (5)

[0048] where R ij is an element of R, whose meaning is the attention magnitude between each word in the current cell i and each word in the target cell j. Subsequently, a max-pooling operation is performed on the matrix R to obtain the attention magnitude between each word in the current cell and the entire semantic information of the target cell. Finally, the attention values of each word in the current cell are added to obtain the correlation degree between the current cell and the target cell. The calculation formula for the correlation degree between two cells can be expressed as:

[0049] e ij = f(U i , U j ) = sum(MaxPooling(R ij )) = sum(MaxPooling((W q U i )(W k U j )) T ), (6)

[0050] If there is a strong correlation between cell i and cell j, then it is expected that e ijThe value should be as large as possible, and conversely, it is desired to be as small as possible. After calculating the correlation degree between the current cell i and all other cells in the same row, it is normalized using the softmax function:

[0051]

[0052] At this time, a ij represents the final correlation degree between cell j and cell i. In the same row or column, the sum of the correlation degrees of all other cells with respect to the current cell i is 1. Subsequently, the information of other cells is weighted and summed according to the obtained correlation degree a ij to obtain the information matrix of the same row (same column) of cell i and

[0053]

[0054] Another main difference between the correlation degree module TCM and the standard graph attention network is that when calculating the correlation degree, the correlation degree of the node with itself is no longer considered. At the same time, after aggregating the information of surrounding nodes, instead of directly updating the information matrix of the current node, it is stored separately first. Finally, after concatenating the cell's own information U i and the information of the same row and column, it is passed into the final classification network. The reason is that the correlation degree module expects the correlation degree between the index data cell and the index classification or index name cell to be as large as possible, and the correlation degree with other data cells to be as small as possible. Therefore, the calculated value of the correlation degree with itself should also be as small as possible. However, when weighted summing the information of all associated cells, due to the too small correlation degree with itself, the updated information of the cell itself will be covered by the other relevant information of the same row and column, resulting in the final classification judgment using the information of other cells and losing its own semantic information. By concatenating the collected information with the original information of itself, not only can the semantic information of the cell itself be retained, but also the corresponding context content is obtained for the current cell, so that the collected information can play an auxiliary judgment role.

[0055] (2) Analysis and calculation of row and column information transfer relationship

[0056] Within the table range, in addition to the information of the same row and column that can play a certain auxiliary role in the final classification of each cell, other information that is not in the same row or column may also have a certain impact. Therefore, the row and column information transfer module TITM is used to interact with the information that is not in the same row or column. The flowchart of the row and column information transfer module TITM is shown in Figure 8 as shown.

[0057] After passing through the correlation module TCM, the cell itself has collected its peer information V row and column information V col , for Figure 7 example, the information "components and materials" at the (1, 2) position, after being calculated by the correlation module, can be obtained by the "smoke generation level" cell at the (1, 5) position and the "luggage rack" cell at the (3, 2) position, and are respectively stored in the row information of the "smoke generation level" cell and the column information of the "luggage rack" cell. If the "SR2" cell at the (3, 5) position wants to obtain the relevant information of the "components and materials" cell at the (1, 2) position, it can obtain it through the cells at its adjacent nodes (1, 5) and (3, 2) positions.

[0058] At this time, use the correlation module again for the (3, 5) cell to obtain the information of the entire third row and the entire fifth column. However, when performing this calculation, the information used is no longer the semantic information U of the cells at the (1, 5) and (3, 2) positions themselves, but V row and V col .

[0059] When calculating with other cells in the same row, since other cells in the same row obtained information through the adjacent nodes in the same row when passing through the correlation module previously, there is no need to use this information again when calculating in the row direction. Only the column information collected by it needs to be processed. Similarly, when calculating with other cells in the column direction, only the row information collected by it needs to be processed.

[0060] The specific calculation formula is as follows:

[0061] Q i =W q U i , (9)

[0062]

[0063]

[0064]

[0065]

[0066]

[0067]

[0068]

[0069]

[0070] Among them, and represents the correlation between the adjacent node j of the current node i and the result obtained by the first association degree module. Finally, the four pieces of information obtained for each cell and the information U of the cell itself i are directly concatenated to obtain which is input into the final fully connected classification network to obtain the index classification result of each cell.

[0071] (3) Process of table information association analysis

[0072] Table information association is to perform information interaction on all cells that are connected to each other in the graph structure, so that cells can obtain more information useful for their final index classification results in the row and column directions. The specific algorithm is as follows:

[0073] 1) Calculate for all cells D in turn. When selecting cell D i , obtain the corresponding adjacency list information:

[0074]

[0075] 2) Take the information U of cell in turn and calculate it with the information U j of cell D i itself to obtain the association degree e i . Subsequently, perform a softmax calculation on all calculated association degrees to obtain the influence degree a ij of each cell j compared to other cells in the same row or the same column as cell i. Finally, use the influence degree a ij = softmax(e ij ) as the weight coefficient to multiply the information of cell j and then sum to obtain the information of the same row ij of cell i and the information of the same column

[0076] 3) Take the information of cell in order again, but use the and calculated in step 2). After calculating its influence degree using the association degree module in the same way, obtain the second-layer information of the same row V i of cell D i row(2) and the second-layer information of the same column V i col(2) ; finally, concatenate all the results obtained from the two calculations with the semantic information of the cell itself to obtain:

[0077]

[0078] (V) Classification and Indicator Information Calculation

[0079] Obtain the output I of the table information association layer i ∈R 15×(5*n) After that, perform dimensional transformation on it to obtain I' i ∈R (5 *n)×15 , then perform max pooling on it to compress it to I'' i ∈R 5*n , and there are a total of N categories for the final indicator classification. Therefore, use a fully connected layer (i.e., a linear classifier) to map it to N classification labels, and the value under each label represents the probability of belonging to that classification.

[0080] In addition to the final category probability of each cell, the final output of the algorithm also includes all the correlation values calculated by the correlation module TCM. Finally, through these correlation values, all the relevant cells within the entire table range can be connected together to piece together the required restored indicator information.

[0081] Technical Features and Advantages of the Present Invention

[0082] 1. Aiming at the characteristic that the size of the table in the standard document is not fixed, a graph network structure is proposed to represent the table.

[0083] 2. To avoid the need to pass through multiple layers of graph networks for data interaction between distant cells when there are too many rows or columns in the table, when using a graph structure to represent the table, other cells in the same row or column of a cell are regarded as its one-hop neighbor nodes, and only two layers of graph network calculations are required to achieve information interaction between any two cells in the entire table. This reduces the complexity of the model, makes it more lightweight, and improves the model training speed.

[0084] 3. Aiming at the characteristic that the influence degree of different cells on other cells is different, the general graph attention network structure is correspondingly adjusted. Q and K matrices are used to learn the correlation degree between two cells. After obtaining the correlation degrees of other cells in the same row and column, in order to avoid losing its own semantic information after the current cell incorporates the information of other cells, two information matrices V row and V col are established to store the information of other cells converged in the same row and column as the context reference of the current cell. And V row and V col are also used as the input of the second layer of graph network, and then the information matrices V row(2) and V col(2), so that when each cell is classified by indicators, it can make judgments by combining the information within the entire table range. Compared with the traditional graph attention network, it can utilize more feature information, which is beneficial to improving the accuracy of table indicator extraction.

[0085] 4. In addition to the final indicator classification result of each cell, the output result of the algorithm also includes the result of the first association degree calculation. The association degree result does not perform gradient descent convergence but is directly used as the output to determine all associated cells in the entire table subsequently, and then the final indicator information can be restored. This facilitates the user to visually view the recognition result and provides a more complete basis for information extraction and stronger interpretability. Brief Description of the Drawings

[0086] Figure 1 It is a flowchart of the method for extracting table indicator information based on the graph attention network of the present invention.

[0087] Figure 2 It is an example of directly converting table information into a graph structure. Among them, (a) is the table and (b) is the corresponding graph structure.

[0088] Figure 3 It is an example of the definition method of one-hop neighbor nodes.

[0089] Figure 4 It is the influence of the whole-column interchange of cells on the graph structure.

[0090] Figure 5 It is an example of cell association.

[0091] Figure 6 It is a flowchart of the association degree module TCM.

[0092] Figure 7 It is an example of row and column information transmission.

[0093] Figure 8 It is a flowchart of the row and column information transmission module.

[0094] Figure 9 It is an example table.

[0095] Figure 10 It is the calculation result of cell association degree. Detailed Embodiment

[0096] The following takes an example to illustrate the detailed embodiment of indicator extraction in the table. For a table as shown in Figure 9 , the goal is to determine which indicator category each cell belongs to. Now, the indicator category of the cell "S3" in the 4th row and 4th column is determined, and the implementation process is as follows.

[0097] (1) Table preprocessing, that is, graph network structuring processing of table information

[0098] Recognize the image table with the help of tools such as OCR, and convert it into an electronic form such as an Excel table or csv, so that the positions of rows and columns and the content of cells can be directly read.

[0099] (2) Cell information encoding

[0100] When converting the text information of each cell into vector information, the size of the finally obtained matrix is word num *embedding size , where embedding size represents the feature dimension of each word vector. In Chinese WordVectors, each word is converted into a fixed length of 300 dimensions, while word num represents the number of words contained in each cell. For cells with more than this number of words, their content needs to be cropped, and for cells with less than this number of words, they need to be filled. If word num is set too large, a large number of cells need to be filled with blank data. On the contrary, if it is set too small, the content of a considerable number of cells will be cropped to varying degrees, resulting in incomplete table information. For this problem, the number of words in each cell is counted after word segmentation.

[0101] In the statistical results, 75% of the cells contain no more than 5 words. Considering that most cells are index data cells, and data cells generally occupy fewer words, which may cause a certain deviation in the statistical results. Therefore, finally, word num is set to 15, that is, the size of the information matrix after word vector conversion for each cell is 15 * 300.

[0102] Perform word vector encoding on the "S3" cell in the table. Its word segmentation result is still "S3". After Word2Vec word vector encoding, the result is input into a bidirectional recurrent neural network (BiLSTM) for semantic feature extraction to obtain the cell semantic information U i .

[0103] (3) Adjacency list analysis

[0104] It is implemented by the adjacency list layer in the network. After obtaining the row adjacency list and column adjacency list of a certain cell, the semantic information U of the cells in the same row or the same column can be obtained by combining word vector encoding j .

[0105] (4) Association degree analysis of the association layer

[0106] Take Ui and U j are input into the table information association layer for association degree analysis. After passing through the association degree module TCM, the association degrees e of other cells in the same row and the same column can be obtained. ij Then, through the softmax function, the influence degree a on other cells is obtained. ij As shown in Figure 10 . It can be found from this that the correlation between the index data cell "S3" and the index sub-category "Insulation materials (sound insulation and / or heat insulation) for side walls, end walls, and floors installed in the area below the bottom edge of the window" is the highest in the row direction. In the column direction, the index name cells "Combustion performance grade" and "Fire protection performance requirements" are found, and the influence of other data cells in the same column on it is relatively small. The visualization result of the association degree calculation shows that through the association degree module, data cells can autonomously find the information of index classification and index name, and then comprehensively judge their own classification information.

[0107] Use the influence degree a ij as the weight coefficient to multiply the information of cell j and then sum to obtain the information V of the same row of cell i i row(1) and the information V of the same column i col(1 ). Then use them as the input of the next-layer TCM, and calculate again to obtain the row and column semantic information V row(2) and V col(2) of the second layer.

[0108] (V) Classification and Association Degree Output

[0109] Transfer the row and column semantic information V i row(1) , V i col(1) , V row(2) , V col(2) and the cell semantic information U i after merging to the classification output layer. After classification by the linear classifier, the index classification result of cell "S3" is obtained. In addition, the information V of the same row of cell "S3" i row(1) and the information V of the same column i col(1) are also used as the outputs of the network.

[0110] The above results effectively demonstrate that in tabular information, other information in the same row and column of a cell can have a certain connection with the current cell, which can play a relatively obvious helpful role in the final classification judgment of each cell. Moreover, other information within the entire table can also provide effective information for the classification result to a certain extent. Compared with other graph attention networks, there are improvements in accuracy, recall, and F1-score, which can make the results of the final indicator extraction more accurate. This shows that the algorithm adopted in the present invention, which converts tabular data into a graph structure and obtains valuable information within the entire table range through the correlation module and the row-column information transmission module, can achieve beneficial effects.

Claims

1. A method for extracting table metric information based on a graph attention network, characterized in that, It is divided into five major parts: (1) Table preprocessing; (2) Cell information encoding; (3) Adjacency list analysis; (4) Table information correlation analysis; (5) Classification and index information calculation; where: The table preprocessing is to perform graph network structuring on the table information to obtain the graph network structure of the table information; then, word vector encoding and adjacency list analysis are respectively performed on the cells in the table. The cell information encoding includes word segmentation and word embedding encoding of the text in the table, and inputting the encoded result into a bidirectional recurrent neural network for semantic feature extraction to obtain the cell semantic information U i ; The adjacent list analysis includes obtaining the row adjacent list and column adjacent list of a certain cell, and combining word vector encoding to obtain the semantic information U of cells in the same row or the same column list. j ; The above-mentioned table information correlation analysis is based on U i and U j to perform table information correlation analysis, including: table information correlation analysis calculation and row-column information transfer relationship analysis calculation; after two-layer correlation calculation, semantic information in the row and column directions of the same row or the same column at two layers is obtained. Among them, the row-column semantic information at the first layer is V row(1) and V col(1) , and the row-column semantic information V row(2) and V col(2) at the second layer. The correlation information at the first layer is used as part of the model output; the row-column semantic information V row(2) and V col(2) at the second layer and the cell semantic information U i are merged and then transferred to the classification output layer for output; For classification and index information calculation, a linear classifier is used for classification to obtain the index classification result of the current cell.

2. The method for extracting tabular index information according to claim 1, wherein The graph network structuring process of the table information mentioned above is to regard the table information with variable size as a graph, each cell in the table as a node in the graph, and adjacent cells as two graph nodes connected to each other, thus converting the information of the entire table into graph information. Adjust the graph structure converted from the table, that is, for each cell in the table, regard all other cells in the same row or the same column as its one-hop neighbors, so that each cell can obtain more table content information on the premise of passing through the shortest possible path. Using this rule to define the graph structure of the table enables any two cells that are not in the same row or the same column to transmit information to each other only through a two-hop distance.

3. The method for extracting table index information according to claim 2, characterized in that, The cell information encoding is to convert the text information of all tables into a computable vector matrix; it includes cell word segmentation and word vector encoding; for cell word segmentation, the jieba word segmentation tool is used to segment the text of the cell; after word segmentation, the Word2Vec method is used to convert each word into the corresponding word vector and splice them; since the information of each cell needs to be converted into a fixed size, each cell intercepts 15 words in length, and the length of each word vector is fixed at 300 dimensions; after cell information encoding, the text information of the entire table is converted into a word vector matrix of {[number of cells]*15*300}; subsequently, the information of each cell passes through the recurrent neural network BiLSTM, enabling the words in each cell to obtain the context relationship within the current cell.

4. The method for extracting table index information according to claim 3, wherein The specific content of the adjacency list analysis is as follows: In the graph network, it is necessary to rely on the adjacency matrix or the adjacency list to represent the structure of the graph; since the graph structure directly connects any cell with other cells in the same row or the same column as its one-hop neighbor nodes, the other cells connected to each cell are calculated according to the row and column positions where each cell is located. The input part of the overall algorithm includes the text information of the table, the number of rows row of the table num and the number of columns column num , and then number each cell in row-major order. For each cell D, calculate the row i and col i where it is located according to its own index number row i = index div column num , col i = index mod column num , By traversing the row coordinates and column coordinates, all adjacent node sets of cell i in the row direction are obtained and all adjacent node sets in the column direction Finally, all the obtained sets of adjacent nodes are concatenated to obtain the row-direction adjacent list Adj row and the column-direction adjacency list Adj col , which are used for the next calculation.

5. The method for extracting table index information according to claim 4, wherein The table information correlation analysis includes table information correlation degree analysis calculation and row and column information transmission relationship analysis calculation. (1) Table information correlation degree analysis calculation: To obtain information from surrounding cells, thereby obtaining supplementary context information to assist in determining the category of its final attribution, an association module TCM is constructed in the graph network to learn the semantic information correlation between two different cells, and based on the information correlation, determine the influence degree of other information in the same row and column on the current cell, determine the weight of the contribution of each cell, and then multiply the obtained weight by the information of other cells in the same row and column and add them up to obtain the text information integrating other cells in the entire row or column; specifically: Construct the query matrix W q and the key matrix W k , where W q is calculated with the semantic information of the currently selected cell, while W k is calculated with other cells in the same row or column as this cell to obtain matrices Q and K: Q i = W q U i , W q Both W k and W are of size d×d, Q and K are of size n×d, and then Q is multiplied by the transposed K to obtain matrix R: Among them, R ij is an element of R, which means the attention magnitude between each word in the current cell i and each word in the target cell j. Subsequently, a max pooling operation is performed on the matrix R to obtain the attention magnitude between each word in the current cell and the entire semantic information of the target cell. Finally, the attention values of each word in the current cell are added together to obtain the correlation degree between the current cell and the target cell. The calculation formula for the correlation degree between two cells is expressed as: e ij = f(U i , U j ) = sum(MaxPooling(R ij )) = sum(MaxPooling((W q U i )(W k U j )) T ), If there is a strong correlation between cell i and cell j, it is desirable that e ij has as large a value as possible, and vice versa, it is desirable that it has as small a value as possible; After calculating the correlation degree of the current cell i with all other cells in the same row , the softmax function is used to normalize it: At this time, a ij represents the association degree between the final cell j and cell i. In the same row or the same column, the sum of the association degrees of all other cells with respect to the current cell i is 1; subsequently, the information of other cells is weighted and summed according to the obtained association degree a ij to obtain the information matrix V of the same row and the same column of cell i i row and V i col : (2) Analysis and calculation of the row-column information transfer relationship; To interact with information that is not in the same row or column and to perform analysis and calculation of the information transfer relationship between rows and columns, a row-column information transfer module (TITM) is constructed; after passing through the correlation module TCM, the cell itself has already collected its row information V row and its column information V col . When calculating with other cells in the same row direction, only the column information it has collected is processed; similarly, when calculating with other cells in the column direction, only the row information it has collected is processed; the specific calculation formula is as follows: Q i = W q U i , Among them, and represent the correlation between the adjacent node j of the current node i and the result calculated by the first association degree module; finally, the four pieces of information V i row , V i col , V i row(2) , V i col(2) obtained for each cell are directly concatenated with the information U i of the cell itself to obtain I = concat([U i , V i row , V i col , V i row(2) , V i col(2) ), which is input into the final fully connected classification network to obtain the index classification result of each cell; (3) Process of table information association analysis Table information association is to perform information interaction on all cells connected to each other in the graph structure, enabling cells to obtain more information useful for their final index classification results in the row and column directions. The specific algorithm is as follows: 1) Calculate for all cells D in sequence, and select cell D i When, obtain the corresponding adjacency list information: 2) Sequentially obtain the information U of the cells and calculate it with the information U of the cell D itself j to obtain the correlation degree e i . Subsequently, perform softmax calculation on all the calculated correlation degrees to obtain the influence degree a of each cell j by the other cells in the same row or the same column compared to the cell i i a = softmax(e ij ). Finally, use the influence degree a ij as the weight coefficient, multiply it by the information of the cell j and then sum them up to obtain the information V of the same row of the cell i ij and the information V of the same column ij row i i col ;​​ 3) Retrieve the cell information in sequence again , but use the V calculated in step 2 i row and V i col . After calculating their influence degrees using the relevance module in the same way, the second-layer peer information V i of cell D i row(2) and the second-layer column information V i col(2) are obtained; finally, the results calculated twice and the semantic information of the cell itself are all concatenated to obtain: I i = Concat([U i , V i row , V i col , V i row(2) , V i col(2) )。 6. The method for extracting table index information according to claim 5, characterized in that The classification and index information calculation is specifically as follows: For I obtained from the correlation analysis of tabular information i ∈R 15×(5*n) , perform dimensionality transformation to obtain I′ i ∈R (5*n)×15 , then perform max pooling on it to compress it to I″ i ∈R 5*n , and finally there are a total of N categories in the index classification category. Use a one-layer fully connected network, that is, a linear classifier, to map it to N classification labels. The value under each label represents the probability of belonging to that classification; In addition, all the correlation values calculated by the association module TCM connect all relevant cells within the entire table range and splice them into the required restored index information.

Images