A Tabular Text Question Answering Method Based on Hierarchical Graph Neural Network
Through the table text question-and-answer method based on hierarchical graph neural network, using sparse search and graph inference technology, the problem of poor inference effect in table text question-and-answer is solved, and efficient information extraction and development of mixed table text data sources are realized. Reasoning improves the accuracy and scalability of the Q&A system.
Patent Information
- Application Number
- CN202210380864.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-04-12
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2042-04-12
AI Technical Summary
Existing Q&A systems do not have the inference effect when processing mixed data sources, especially in table text Q&A scenarios. Traditional methods such as HYBRIDER and DocHopper have low inference accuracy and redundant processes, making it difficult to effectively utilize semi-structured tables and Information on unstructured text.
The table text question-and-answer method based on hierarchical graph neural network is used to initially locate the answers through sparse searchers and pre-trained language models, and use the hierarchical graph neural network model to perform graph inference on questions, cells and short essays, and combine multi-task loss function to provide accurate answers. Positioning includes steps such as key information retrieval, table row selection, long text encoding and hierarchical graph reasoning.
It realizes the rapid and accurate capture of structured information of tabular data, improves the inference effect of mixed table text data sources, can be applied in a variety of question-and-answer scenarios, and has strong scalability.
Smart Images

Figure CN114969279B_ABST
Abstract
Description
Technical Field
[0001] The invention relates to a table text question-answering method based on a hierarchical graph neural network, and belongs to the technical field of Internet and artificial intelligence. Background Art
[0002] With the continuous development and progress of related technologies such as big data, network engineering, and communication engineering, the dissemination of information has become easier and easier, resulting in an exponential growth in the amount of various complex information on the Internet. According to statistics, most of the information on the Internet exists in the form of text, but there is also a lot of information stored in semi-structured tables. Compared with unstructured text, tables have more rigorous information aggregation, but the amount of information in each cell in the table is often small, usually representing a phrase or a value. If the table information and text information can be combined, the amount of information can be enriched while maintaining the structured features better. For example, most of the cells in the table on Wikipedia are linked to related encyclopedia entities as a supplement to the table information. How to use natural language processing technology to perform machine reading comprehension on mixed information of tables and text is a difficult problem with great application value.
[0003] At present, the tasks of traditional machine reading comprehension are mainly divided into: cloze, multiple choice, common sense question and answer, and fragment extraction. Among them, fragment extraction refers to extracting text fragments from the material as answers from a given expected range through reasoning and analysis. The most typical application of fragment extraction tasks is the question-answering system, which allows people to ask questions that conform to the form of natural language, and then directly returns the answer after in-depth question analysis and knowledge reasoning. The question-answering system can help users quickly focus on key information from the corpus, significantly improving people's efficiency in processing information. Due to the reduction of user processing time and the convenience of voice interaction, the question-answering system has been widely used in the fields of intelligent customer service, intelligent retrieval, and IOT smart home. Although more and more question-answering related algorithms and systems have been proposed, there are still many challenges in this field that need to be solved. First, in most data sets, the data source is limited to one type of information, such as unstructured text, semi-structured tables, or knowledge-based knowledge graphs. However, different types of data are usually complementary to each other.
[0004] At present, hybrid data source reasoning scenarios consisting of unstructured text and semi-structured tables are gaining more and more attention. However, the processes of most question-answering systems are redundant and complicated, or the reasoning effect is poor. A tabular text reasoning scheme called HYBRIDER, which mainly consists of three models in the reasoning stage: sorting model, multi-hop model, and reading comprehension model. Although HYBRIDER is well designed, its reasoning effect is still not ideal and the reasoning accuracy is low. There is another scheme called DocHopper, which extends the existing attention-based method through a new mechanism to update questions in multi-hop iterations. DocHopper performs well in multiple scenarios, but DocHopper has limited reasoning effect on tabular text question-answering scenarios because DocHopper can only be applied to unstructured documents. Summary of the invention
[0005] Purpose of the invention: In view of the problems and shortcomings existing in the prior art, the present invention proposes a table text question-answering method based on a hierarchical graph neural network, which can quickly analyze and reason about complex and redundant table text information to obtain the answer corresponding to the question.
[0006] Technical solution: To achieve the above invention purpose, the present invention provides a table text question answering method based on hierarchical graph neural network, which first preliminarily locates the answer to the question, selects the table row where the answer is most likely to exist, and then uses graph reasoning to accurately locate the answer. The method mainly includes five steps, as follows:
[0007] (1) Key information retrieval: Use a sparse search retriever and a longest substring matcher to find the supporting evidence that is most likely to be relevant to the answer in the table and the passage, marked as SE (Support Evidence). The retriever calculates the similarity between the cells and the sentences in the passage and the question, and selects relevant evidence information with a similarity less than 0.8 for marking.
[0008] (2) Table row selection: Using TaPas as the table encoding model, the table rows containing SE information are pre-encoded with TaPas, and then all the retrieved SE information is filled into the table to obtain the vector representation of the entire table through a table row selector.
[0009] (3) Long text encoding: Use a long encoding model to encode the table row and the external link text of the table row to obtain the feature vector representation of the input.
[0010] (4) Hierarchical graph reasoning: The present invention designs and constructs a hierarchical graph neural network model called row-HGN, which is used to model three types of nodes (questions, cells, and passages), and then performs graph reasoning between the three types of nodes to capture semi-structured information.
[0011] (5) Multi-task answer extraction: A mixed loss is used as a function of model back-propagation to finally locate the answer through three sub-tasks: cell selection, answer location selection, and answer position location.
[0012] Compared with the prior art, the beneficial effects of the present invention are as follows:
[0013] 1. The present invention designs a row selection scheme including a sparse retriever and a pre-trained language model, which solves the problem of initial location of answers for mixed heterogeneous data and can quickly find the table row that is most likely to contain the answer. Compared with the prior art, the present invention can quickly and accurately capture the structured information of tabular data, thereby making inferences.
[0014] 2. This paper proposes a hierarchical graph neural network for tabular text data to perform graph reasoning on questions, cells, and passages. In addition, three subtasks are constructed to accurately locate the answers. Compared with traditional methods, this paper can better reason about the mixed data source information of tabular text and accurately extract the answer corresponding to the question from the answer row.
[0015] 3. The table row-based hierarchical graph neural network proposed in the present invention can be applied to a variety of table- and text-oriented question-answering scenarios, as long as the nodes are replaced, and has strong scalability. BRIEF DESCRIPTION OF THE DRAWINGS
[0016] Figure 1 The figure is a flow chart of a method according to an embodiment of the present invention.
[0017] Figure 2 This is a table row selection diagram according to an embodiment of the present invention.
[0018] Figure 3 This is a table row reading comprehension diagram for an embodiment of the present invention. DETAILED DESCRIPTION
[0019] The technical solution provided by the present invention will be described in detail below in conjunction with specific embodiments. It should be understood that the following specific implementation methods are only used to illustrate the present invention and are not used to limit the scope of the present invention.
[0020] Embodiment 1: The present invention provides a table text question answering method based on a hierarchical graph neural network, the method comprising the following steps:
[0021] Step 1: Key information retrieval,
[0022] The present invention completes the table row metadata selection task through TF-IDF word frequency retrieval and TaPas, uses TF-IDF to perform coarse recall of question-related information, aiming to reduce the interference of redundant information and noise on the subsequent reasoning process, and then uses Tapas to encode the table containing key evidence, aiming to enable reasoning between different table rows, and finally uses a classifier to select the table row metadata where the answer is most likely to exist.
[0023] The main goal of the table row selection scheme is to select the table row where the answer is most likely to exist. The present invention uses a TF-IDF searcher and a longest substring matcher to find the supporting evidence that is most likely to be related to the answer in the table and the short article, marked as SE (Support Evidence). The searcher will calculate the similarity between the cell and the short article sentence and the question, and will select the relevant evidence information with a similarity less than 0.8 and mark it as SE. That is to say, SE may come from the cell directly related to the question or from a sentence in the short article. An SE unit mainly includes three parts, namely: content, location, and description, which is the value text of the cell, location is the coordinate of this cell, and Description is set to empty; if SE comes from passage, then content is the title of passage, location is the position corresponding to the cell linked to this passage, and Description is the sentence related to the question retrieved in this passage. In the row selection stage of the present invention, as shown in formula (1), the present invention treats all SEs as auxiliary information. After inserting them into the corresponding cells in the table, some cells now contain not only the text of the cell itself, but finally the entire table is sent to the pre-trained language model to decide which row is most likely to contain the answer.
[0024] Cell′ i,j =Concatenate(Cell i,j ,SE[i,j]) (1)
[0025] Step 2: Table row selection,
[0026] The present invention uses TaPas as a table encoding model. The TaPas model adds some position features unique to tabular data on top of the traditional Bert model, including: Position ID, Segment ID, Column ID, Row ID, and Rank ID. Compared with traditional absolute position features, these table-level position features can more effectively capture semi-structured information. The detailed structure of TaPas has been described above and will not be described here. Through the pre-retrieval of TF-IDF in the previous step, the partial SE associated with the question is obtained, and after the SE is filled into the corresponding cells in the table, the present invention pre-encodes the new table containing SE information with TaPas, and then fills all the retrieved SP information into the table to obtain the vector representation of the entire table, denoted as TR (Table Representation).
[0027] TR=TaPas([cell′ 0,0 ,…,cell′ i,j ,…,cell′ m,n ]) (2)
[0028] Treat a table row (including all passages linked to the cells in this row) as metadata to perform table row reading comprehension tasks downstream.
[0029] Row i =MLP(Pooling(TR[start i ∶end i ])) (3)
[0030] Pooling means pooling, start i and end i Indicates the start and end position of each row, and then Row i The vector matrix is sent to the classifier composed of two layers of MLP, and the row with the most likely answer is selected according to the size of logits of each row.
[0031] Row selected =max(softmax(row i ) (4)
[0032] The present invention selects the row with the highest probability as the answer row and serves as the input for the next stage.
[0033] Step 3: Long text encoding,
[0034] Due to the uncertainty of the answer position, the answer needs to be found in all cells in the answer row and all the short texts linked to these cells. But in most cases, for language models such as Bert, the input length is limited to 512, which is not enough for the length of most rows of text. A common solution is to divide the long text into different blocks of length 512, then process each block separately, and then combine the results of each block with the model of the specific task, but this is time-consuming, and the quality of the block division may have a greater impact on the answer. The system of the present invention uses Longformer as the second-stage encoder. Traditional Bert uses full attention, that is, all input tokens will be calculated with all other tokens for attention, which is more suitable for situations where the text length is not very long. If the text length is very long, then O(n 2 ) is very computationally resource-intensive. Longformer proposes an attention mode that combines sliding window, extended sliding window, and global attention. Different attention methods are used for different text segments to maximize the reliability of text information interaction while reducing the amount of calculation. The present invention sets the question field to full attention and the information part to sliding window attention, because the question should have an impact on all text information, and only the closest parts of text tokens will affect each other. The amount of information shared between the first sentence and the last sentence of a paragraph is far less than the amount of information shared between two adjacent sentences.
[0035] The present invention converts the table row data into sentences in natural language form, for example: "Header1 is A, Header is B", "header" represents the column name, A and B are the contents of the cell. The present invention splices these three parts in the order of question, cell, and short article. Then different nodes are connected in the form of edges to capture structural information.
[0036] E=LongFormer(Q,C,P) (5)
[0037] Step 4: Hierarchical graph reasoning
[0038] After the long text encoding in the previous step, the present invention obtains the embedding of each token of the answer row metadata. The present invention constructs a hierarchical graph neural network model called row-HGN, uses three types of nodes (questions, cells, and short texts) in row-HGN for modeling, and then performs graph reasoning. The construction methods of the three types of nodes are as follows:
[0039]
[0040]
[0041] In the above formula, E∈R l*d represents the feature vector obtained by Longformer encoding in the previous step, l is the input length (set to 1024 in the present invention), d is the dimension of the embedded vector, and Indicates the start and end position of the i-th cell, and represents the start and end position of the i-th passage, mean(·) represents the average pooling operation, and finally we get Q: the feature vector space of the problem, The eigenvector space of the cell, The feature vector space of the short text associated with the cell in this row. Let G = {Q, C, P} be represented as the node set of the row-HGH of the present invention. In order to better capture the structural information of the table row, the present invention defines two types of edges: (1) the edge between the question node and the cell node; (2) the edge between the cell node and the short text node connected to it. This construction reduces the complexity of the hierarchical graph on the one hand, and captures the structure of the table row and the external link short text as much as possible on the other hand. The present invention uses a gating mechanism to determine how much information to propagate to the next step. Aggregate information from the neighbors of each node in the following way:
[0042]
[0043] Among them, 0≤l≤L represents the layer, N i represents the neighbor of node i, node, ‖*‖ represents the size of the node set, f1 is a transformation function implemented by MLP, is the feature vector representation of node j at layer l. The present invention solves the smoothing problem in the reasoning process by adding a gating mechanism.
[0044] Step 5: Multi-task answer extraction
[0045] After graph reasoning, the present invention obtains the feature vector of the graph node G′={Q′,C′,P′},G′∈R n*d ,n=1+n c +n p,This step constructs three different subtasks to finally locate the final location of the answer, namely: answer classification, cell selection, and span location. First, for the first subtask, answer classification, the present invention needs to determine whether the answer is in the cell or in the passage. This subtask is a binary classification task ("In Table" or "In passage"). The information based on is the embedding of the Q′ node obtained after graph reasoning. The process is described by the formula:
[0046]
[0047] Secondly, for the answers in the cells, the present invention needs to perform a cell selection subtask, which is used to determine which cell in the cell row is most likely to contain the answer, because it is unknown which cell the answer is in. This step is described by the formula:
[0048]
[0049] Finally, for the answers in the paragraph, it is necessary to accurately extract the start and end positions of the answer fragments. The present invention concatenates P′ obtained by graph reasoning and the initial feature E obtained by long coding, and then calculates the start and end positions of the answer as follows:
[0050]
[0051]
[0052] OutputLayer is a hybrid output layer, which is mainly composed of two linear layers and one BertBorm layer. Finally, the present invention uses a hybrid loss function as the function of model back propagation. The total loss combines the losses of the three subtasks of cell selection, answer location selection and answer location location, and is calculated as follows:
[0053] L mix =αL location +βL cell +γL span (15)
[0054] Among them, α, β, γ are hyperparameters, L location , L cell , L span Corresponding to the loss values of the three subtasks respectively.
[0055] In summary, the present invention proposes a two-stage table text question answering reasoning method. First, a sparse searcher and a table row selector are used to select the table row where the answer is most likely to exist. Secondly, the present invention proposes a hierarchical graph neural network model based on table rows to model questions, cells, and essays to capture the semi-structured and textual information of table and text data. Finally, the answer to the question is obtained through multi-task solving.
[0056] It should be understood that these embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention. After reading the present invention, various equivalent modifications of the present invention by those skilled in the art all fall within the scope defined by the claims attached to this application.
Claims
1. A table text question answering method based on hierarchical graph neural network, characterized in that: The steps include: (1) Key information retrieval: Use a sparse search retriever and a longest substring matcher to find the supporting evidence that is most likely to be relevant to the answer in the table and the passage, marked as SE, Support Evidence. The retriever calculates the similarity between the cell and the sentence in the passage and the question, and selects relevant evidence information with a similarity less than 0.8 for labeling; (2) Table row selection: Using TaPas as the table encoding model, the table rows containing SE information are pre-encoded with TaPas, and then all the retrieved SE information is filled into the table to obtain the vector representation of the entire table through a table row selector; (3) Long text encoding: Use a long encoding model to encode the table row and the external link text of the table row to obtain the input feature vector representation; (4) Hierarchical graph reasoning: This paper designs and constructs a hierarchical graph neural network model called row-HGN, which is used to model three types of nodes: questions, cells, and passages. Then, graph reasoning is performed between the three types of nodes to capture semi-structured information. (5) Multi-task answer extraction: A mixed loss is used as a function of model back propagation to finally locate the answer through three sub-tasks: cell selection, answer location selection, and answer position location. The multi-task answer extraction in step (5) is as follows: After graph reasoning, the present invention obtains the feature vector of the graph node G′={Q′,C′,P′},G′∈R n*d ,n=1+n c +n p , Q′, C′, and P′ are the feature vectors of the question, cell, and passage nodes obtained after graph reasoning. This step constructs three different subtasks to finally locate the final position of the answer, namely: answer classification, cell selection, and span location. First, for the first subtask, the information based on classification is the feature vector of the Q′ node obtained after graph reasoning. The process is described by the formula: Second, for the answer in the cell, a cell selection subtask needs to be performed, described by the formula: Finally, for the answer in the paragraph, it is necessary to accurately extract the start and end positions of the answer fragment, concatenate P′ obtained by graph reasoning and the initial feature E obtained by long encoding, and then calculate the start and end positions of the answer as follows: In the above formula, OutputLayer is a hybrid output layer, which is mainly composed of two linear layers and one BertBorm layer. Finally, the present invention uses a hybrid loss function as the function of model back propagation. The total L mix The loss of the three subtasks of cell selection, answer location selection, and answer location location is combined and calculated as follows: L mix =βL location +αL cell +γL span (15) Among them, α, β, γ are hyperparameters, L location , L cell , L span Corresponding to the loss values of the three subtasks respectively.
2. The table text question answering method based on hierarchical graph neural network according to claim 1 is characterized in that: The TF-IDF searcher selects relevant evidence information with a similarity less than 0.8 and marks it as SE, and fills it into the table.
3. The table text question answering method based on hierarchical graph neural network according to claim 1 is characterized in that: In step (2), the table row selection model only selects table rows containing SE information for sorting. The process is as follows: Row i =MLP(Pooling(TR[start i ∶end i ])) start i and end i Indicates the start and end position of each row, and then Row i The vector matrix is sent to the classifier composed of two layers of MLP. According to the size of logits of each row, the row with the most likely answer is selected. MLP is a multi-layer perceptron, and Pooling represents the pooling layer. The table row with the largest probability is selected as the answer row. The calculation is as follows: Row selected =max(softmax(row i ) Among them, softmax is the logistic regression function and max is the maximum value function.
4. The table text question answering method based on hierarchical graph neural network according to claim 1 is characterized in that: The long text encoding in step (3) converts the table row data into sentences in natural language form, concatenates them in the order of questions, cells, and short texts, and then connects different nodes in the form of edges to capture structural information; E=LongFormer(Q,C,P) Among them, Q, C, and P represent questions, cells, and short text inputs respectively, and LongFormer() means using the Longformer model to encode long texts.
5. The table text question answering method based on hierarchical graph neural network according to claim 1 is characterized in that: In step (4), the table row graph neural network uses a gating mechanism to decide how much information to propagate to the next step. First, it aggregates information from each node’s neighbors in the following way: Among them, 0≤l≤L represents the layer, N i represents the neighbor of node i, node, ∥*∥ represents the size of the node set, f1 is a transformation function implemented by MLP, is the feature vector representation of node j at layer l. The present invention solves the smoothing problem in the reasoning process by adding a gating mechanism, which is calculated as follows: Among them, sigmoid and tanh are activation functions, and f2 is implemented by a multi-layer perceptron. is the feature vector of node i in the l+1th layer.
