Table text numerical reasoning question and answer method based on joint modeling

Abstract

The invention discloses a table and text numerical reasoning question-answering method based on joint modeling, which comprises the following steps: receiving a document containing a text and a table and a question proposed by a user, retrieving support evidence according to the question through a heterogeneous evidence retrieval module, reserving the two-dimensional structural integrity of the table, avoiding information loss caused by serialization, and obtaining a question-answering result on the basis of the retrieved evidence. Constructing a heterogeneous graph, encoding the heterogeneous graph by using a relationship perception graph neural network, capturing interaction information between table and text data and a hierarchical dependency relationship in a table, and generating a mathematical expression or a numerical answer for answering a question according to learned node representation, according to the table and text numerical reasoning question-answering method based on joint modeling, structured information of the table and semantic information of the text can be deeply integrated, an accurate mathematical expression or numerical answer is generated, and the accuracy, interpretability and applicability of a question-answering system are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular to a table-text numerical reasoning question-answering method based on joint modeling. Background Art

[0002] Question answering systems for numerical reasoning based on tables and text are an important research area in natural language processing (NLP). Their goal is to solve complex numerical reasoning problems by combining tables and text. In real-world applications such as financial analysis, scientific computing, intelligent customer service, and educational assistance, users often need to extract information from tabular data and related text descriptions and perform numerical calculations to obtain answers.

[0003] However, traditional question-answering systems typically serialize tabular data into linear text to adapt to the input format of pre-trained language models. While this approach simplifies the data processing process, it inevitably results in the loss of the two-dimensional structure and hierarchical semantic information of the table, making it difficult for the model to accurately capture the complex semantic associations between table elements and text content. Secondly, during multi-step numerical reasoning, existing technologies are insufficiently capable of modeling numerical relationships. They rely on the implicit understanding of pre-trained language models and lack explicit numerical representation enhancement mechanisms, resulting in poor accuracy and robustness when handling numerical comparisons and complex calculation tasks.

[0004] For example, the FinQANet method simplifies the processing flow by converting tables into text, but does not consider the hierarchical structure information of the table, resulting in missing information in the generated table description and the inability to effectively support numerical comparison reasoning. The MT2Net method converts table content into a table description in natural language text through a template, but this conversion also loses the hierarchical structure information of the table, affecting the overall effectiveness of the question-answering system. Although the NAPG method alleviates the error accumulation problem through a non-autoregressive decoder, it still does not optimize structural modeling and numerical relationships. The TaCube method assists the pre-trained model in answering numerical reasoning questions by pre-generating table information, but this method increases the overhead of data preprocessing and cannot cover all possible computational situations.

[0005] In summary, although the existing technology has made certain progress in the field of table-text numerical reasoning question answering, it still has problems such as loss of table structure information and insufficient numerical relationship modeling, which makes it difficult to meet the needs of complex data-driven intelligent question answering systems. Therefore, the present invention proposes a table-text numerical reasoning question answering method based on joint modeling to solve the above problems. Summary of the Invention

[0006] In view of the above-mentioned defects in the prior art, the purpose of the present invention is to provide a table-text numerical reasoning question-answering method based on joint modeling.

[0007] To achieve the above objectives, the present invention provides a table-text numerical reasoning question-answering method based on joint modeling, comprising the following steps:

[0008] Step 1: Receive a document containing text and tables and questions raised by the user;

[0009] Step 2: Using the heterogeneous evidence retrieval module, support evidence is retrieved based on the question, preserving the two-dimensional structural integrity of the table and avoiding information loss caused by serialization;

[0010] Step 3: Based on the retrieved evidence, a heterogeneous graph is constructed. The heterogeneous graph includes question nodes, text paragraph nodes, table cell nodes, and table structure information nodes. The edges between the nodes represent the semantic associations between them.

[0011] Step 4: Encode the heterogeneous graph using a relation-aware graph neural network to capture the interaction information between table and text data as well as the hierarchical dependencies within the table;

[0012] Step 5: Generate mathematical expressions or numerical answers to answer questions based on the learned node representations.

[0013] Furthermore, the heterogeneous evidence retrieval module includes: for text data, through an intensive retrieval method, using a pre-trained language model to encode questions and text paragraphs, calculating similarity scores, and selecting several paragraphs with the highest scores as candidate text evidence; for tabular data, adopting a structured retrieval method, using a table encoder to encode questions and tables, predicting the relevance between questions and tables through a classifier, and selecting tables with high relevance as candidate tabular evidence; introducing the idea of contrastive learning in text retrieval, and using a batch negative sampling strategy to optimize model performance; in table retrieval, using a multi-layer perceptron (MLP) to classify the output of the table encoder and output the predicted probability of the relevance between the question and the table.

[0014] Furthermore, the construction of the heterogeneous graph includes the following node types: question nodes, which represent the semantic information of the questions raised by the user; text paragraph nodes, which represent the semantic information of the retrieved text evidence; numerical nodes, which represent all numbers appearing in the questions, texts and tables; row header nodes and column header nodes, which represent the row and column structure information of the table; hierarchical header nodes, which represent the nested hierarchical structure information in the table; date nodes, which represent the time information appearing in the questions, texts and tables; table nodes, which represent the global information of the entire table; special character or constant nodes, which represent special characters or constants used for expression generation.

[0015] Furthermore, the construction of the heterogeneous graph includes the following types: internal relationship edges in a table, such as row header to value, column header to value, table to row header, etc.; internal relationship edges in a paragraph or question, such as the relationship between a paragraph or question and the value or date it contains; relationship edges between question and document, such as question to paragraph, question to table; relationship edges between paragraph and table, such as paragraph to table; relationship edges between tables, such as between tables with the same row / column headers; cross-modal relationship edges, such as the association relationship between text paragraphs and table cells.

[0016] Furthermore, the relationship-aware graph neural network includes: node initialization, designing specific representation methods according to different node types, generating initial features of nodes by aggregating sub-word information or title information; message passing mechanism, calculating attention scores between nodes through the relational attention mechanism, aggregating messages of neighboring nodes and edges to update node representations; layer normalization and residual connection, preventing the problem of similar node representations caused by too many layers of the graph neural network; node classification task, calculating each numerical node as evidence, and introducing a learnable threshold to adaptively screen evidence nodes.

[0017] Furthermore, the method further includes a numerical representation enhancement module for explicitly modeling the quantitative relationship between numerical values, including: multi-granularity semantic encoding, which captures the magnitude information and contextual relationship of numerical values through absolute semantic channels and relative semantic channels respectively; numerical relationship recognition loss function, which optimizes the size relationship and extreme value relationship between numerical values, and improves the model's ability to understand numerical comparison; binary relationship loss, which measures the size relationship between any two numerical values, and regards the recognition of numerical pair relationships as a three-classification task; extreme value recognition loss, which combines cross-entropy loss and ranking loss to identify the maximum and minimum values, and maintain a clear interval between them and other numerical values.

[0018] Furthermore, the multi-granularity semantic encoding includes: logarithmic bucket encoding, mapping the numerical value to the logarithmic space and then discretizing it to solve the information imbalance problem caused by too large or too small numerical range; mapping the logarithmic encoded numerical value to a fixed-dimensional space to capture the relative position information of the numerical value; sub-word attention pooling to obtain the contextual semantic representation of the numerical value; gated fusion mechanism to adaptively adjust the contribution of absolute semantics and relative semantics according to task requirements.

[0019] Furthermore, the method further includes a semantic fusion module for fusing numerical relationships and table structure information, including: a gating mechanism that dynamically adjusts the fusion weight of numerical information and graph structure information according to the needs of the node; adaptive fusion that obtains the final node representation through weighted summation and optimizes the information selection of the model in different reasoning steps; in the expression generation stage, the contextual information of the input and decoding history is fused through the attention mechanism, the score of each character is calculated, and the character with the highest score is selected as the prediction result.

[0020] Furthermore, the expression generation step includes: using a decoder, such as a long short-term memory network (LSTM), to generate a computable expression based on the learned node representation; the decoding space for expression generation includes numerical nodes, special characters or constants, and memory characters; fusing the contextual information of the input and decoding history through the attention mechanism, calculating the score of each character and selecting the character with the highest score as the prediction result; during the training process, optimizing the expression generation ability of the model through cross-entropy loss, and generating accurate mathematical expressions or numerical answers by deeply integrating the structured information of the table with the semantic information of the text, which significantly improves the accuracy, interpretability and applicability of the question-answering system, and solves the problems of table structure information loss and insufficient numerical relationship modeling in the prior art; explicitly retaining and utilizing the two-dimensional structure and hierarchical semantic information of the table through heterogeneous graph modeling and relationship-aware graph neural network; explicitly retaining and utilizing the two-dimensional structure and hierarchical semantic information of the table through the numerical representation enhancement module. Modeling the quantitative relationship between numerical values improves the model's ability to understand numerical reasoning; through the semantic fusion module, dynamically integrate numerical information and structural information, and optimize the model's information selection in different reasoning steps. The method also includes a step of training the model, and the training data includes labeled questions, texts, tables, and corresponding mathematical expressions or numerical answers. By optimizing the parameters of the heterogeneous evidence retrieval module, heterogeneous graph construction module, graph neural network encoding module, numerical representation enhancement module, semantic fusion module, and expression generation module, the model achieves optimal performance on the training data; during the training process, the performance of the text retrieval module is optimized through comparative learning; the performance of the table retrieval module is optimized through cross entropy loss and ranking loss; the performance of the graph neural network encoding module and the expression generation module are optimized through node classification loss and expression generation loss; and the performance of the numerical representation enhancement module is optimized through numerical relationship recognition loss.

[0021] Furthermore, the method further includes a dynamic decoding strategy based on reinforcement learning to optimize the decoding path selection during the expression generation process. Specifically, it includes: designing a reinforcement learning agent to select the optimal decoding action at each step of expression generation; dynamically adjusting the decoding strategy based on the accuracy and complexity of the generated expression through a reward mechanism; combining supervised learning and reinforcement learning during training to enable the model to learn the optimal decoding path and reduce the probability of error generation; in multi-step numerical reasoning tasks, the dynamic decoding strategy adaptively adjusts the numerical comparison and operation order during the decoding process to improve the generalization ability of the model. The method further includes an adaptive reasoning path optimization mechanism to dynamically adjust the information flow and computation path during the reasoning process to improve reasoning efficiency and accuracy. Specifically, it includes: introducing reasoning path nodes in the heterogeneous graph to represent the computation path during the reasoning process; dynamically selecting the optimal reasoning path through a path attention mechanism to reduce invalid calculations and error accumulation; adaptively adjusting the depth and breadth of the reasoning path based on the complexity of the problem and the structural characteristics of the data; and dynamically adjusting the order and method of using intermediate results in multi-step reasoning tasks through a path optimization mechanism to improve the robustness of reasoning.

[0022] Compared with the prior art, the present invention has the following beneficial effects:

[0023] 1. The present invention explicitly retains and utilizes the two-dimensional structure and hierarchical semantic information of the table through heterogeneous graph modeling and relationship-aware graph neural network, while integrating the semantic information of the text to generate accurate mathematical expressions or numerical answers. The numerical representation enhancement module explicitly models the quantitative relationship between numerical values, improves the model's ability to understand numerical reasoning, and reduces errors caused by numerical comparisons and complex calculations.

[0024] 2. The semantic fusion module of the present invention dynamically integrates numerical information and table structure information, optimizes the information selection of the model in different reasoning steps, makes the reasoning process more transparent and explainable, calculates each numerical node as evidence through the node classification task, and introduces a learnable threshold to adaptively screen evidence nodes to ensure the rationality of the reasoning process.

[0025] 3. The dynamic decoding strategy based on reinforcement learning in this invention optimizes the decoding path selection in the expression generation process, improves the generalization ability and robustness of the model, and can handle more complex numerical reasoning tasks. It dynamically adjusts the information flow and calculation path in the reasoning process through the adaptive reasoning path optimization mechanism, reduces invalid calculations and error accumulation, and improves reasoning efficiency and accuracy.

[0026] 4. The present invention optimizes the performance of the text retrieval module through comparative learning, optimizes the performance of the table retrieval module through cross entropy loss and ranking loss, optimizes the performance of the graph neural network encoding module and the expression generation module through node classification loss and expression generation loss, and optimizes the performance of the numerical representation enhancement module through numerical relationship recognition loss. The multi-granularity semantic encoding of the present invention captures the magnitude information and contextual relationship of the numerical value through absolute semantic channels and relative semantic channels respectively, solves the information imbalance problem caused by too large or too small numerical range, and improves the robustness of the model.

[0027] 5. The table-text numerical reasoning question-answering method based on joint modeling of the present invention can significantly improve the performance of the question-answering system, meet the needs of complex data-driven intelligent question-answering systems, and provide innovative solutions for applications in multiple fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the solutions in the present invention, a brief introduction is given below to the drawings required for use in describing the embodiments of the present invention. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0029] Figure 1 It is a schematic diagram of the process provided by the present invention;

[0030] Figure 2 It is a framework diagram provided by the present invention; DETAILED DESCRIPTION

[0031] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings so that the advantages and features of the present invention can be more easily understood by those skilled in the art, thereby making a clearer and more precise definition of the scope of protection of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without making any creative work are within the scope of protection of the present invention.

[0032] The terms "including," "having," and any variations thereof in the present specification, claims, and accompanying drawings are intended to cover non-exclusive inclusions. The terms "first," "second," and the like in the present specification, claims, and accompanying drawings are used to distinguish between different items, not to describe a specific order.

[0033] See also Figure 1-2 The table-text numerical reasoning question answering method based on joint modeling includes the following steps:

[0034] Step 1: Receive a document containing text and tables and questions raised by the user;

[0035] Step 2: Use the heterogeneous evidence retrieval module to retrieve supporting evidence based on the question, preserving the two-dimensional structural integrity of the table and avoiding information loss caused by serialization;

[0036] Step 3: Based on the retrieved evidence, a heterogeneous graph is constructed. The heterogeneous graph includes question nodes, text paragraph nodes, table cell nodes, and table structure information nodes. The edges between nodes represent the semantic associations between them.

[0037] Step 4: Use the relationship-aware graph neural network to encode the heterogeneous graph, capturing the interaction information between table and text data as well as the hierarchical dependencies within the table;

[0038] Step 5: Generate mathematical expressions or numerical answers to answer questions based on the learned node representations.

[0039] As an improvement to the above technical solution, the heterogeneous evidence retrieval module includes: for text data, through an intensive retrieval method, a pre-trained language model is used to encode questions and text paragraphs, similarity scores are calculated, and several paragraphs with the highest scores are selected as candidate text evidence; for tabular data, a structured retrieval method is adopted, and a table encoder is used to encode questions and tables. The relevance between questions and tables is predicted by a classifier, and tables with high relevance are selected as candidate tabular evidence; the idea of contrastive learning is introduced in text retrieval, and the within-batch negative sampling strategy is used to optimize model performance; in table retrieval, the output of the table encoder is classified by a multi-layer perceptron (MLP), and the predicted probability of the relevance between questions and tables is output.

[0040] As an improvement to the above technical solution, the construction of the heterogeneous graph includes the following node types: question nodes, which represent the semantic information of the questions raised by the user; text paragraph nodes, which represent the semantic information of the retrieved text evidence; numerical nodes, which represent all numbers appearing in the questions, texts and tables; row header nodes and column header nodes, which represent the row and column structure information of the table; hierarchical header nodes, which represent the nested hierarchical structure information in the table; date nodes, which represent the time information appearing in the questions, texts and tables; table nodes, which represent the global information of the entire table; special character or constant nodes, which represent special characters or constants used for expression generation.

[0041] As an improvement to the above technical solution, the construction of heterogeneous graphs includes the following types: internal relationship edges in tables, such as row header to value, column header to value, table to row header, etc.; internal relationship edges in paragraphs or questions, such as the relationship between a paragraph or question and the value or date it contains; relationship edges between questions and documents, such as question to paragraph, question to table; relationship edges between paragraphs and tables, such as paragraph to table; relationship edges between tables, such as between tables with the same row / column headers; cross-modal relationship edges, such as the association relationship between text paragraphs and table cells.

[0042] As an improvement of the above technical solution, the relationship-aware graph neural network includes: node initialization, designing specific representation methods according to different node types, generating initial features of nodes by aggregating sub-word information or title information; message passing mechanism, calculating attention scores between nodes through the relational attention mechanism, aggregating messages of neighboring nodes and edges to update node representations; layer normalization and residual connection, preventing the problem of similar node representations caused by too many layers of graph neural networks; node classification task, calculating each numerical node as evidence, and introducing a learnable threshold to adaptively screen evidence nodes.

[0043] As an improvement of the above technical solution, the method further includes a numerical representation enhancement module for explicitly modeling the quantitative relationship between numerical values, including: multi-granularity semantic encoding, which captures the magnitude information and contextual relationship of numerical values through absolute semantic channels and relative semantic channels respectively; numerical relationship recognition loss function, which optimizes the size relationship and extreme value relationship between numerical values, and improves the model's ability to understand numerical comparison; binary relationship loss, which measures the size relationship between any two numerical values and regards the recognition of numerical pair relationships as a three-classification task; extreme value recognition loss, which combines cross-entropy loss and ranking loss to identify the maximum and minimum values and maintain a clear interval between them and other numerical values.

[0044] As an improvement to the above technical solution, multi-granularity semantic coding includes: logarithmic bucket encoding, which maps the numerical value to the logarithmic space and then discretizes it to solve the information imbalance problem caused by too large or too small numerical range; mapping the logarithmically encoded numerical value to a fixed-dimensional space to capture the relative position information of the numerical value; sub-word attention pooling to obtain the contextual semantic representation of the numerical value; and a gated fusion mechanism to adaptively adjust the contribution of absolute semantics and relative semantics according to task requirements.

[0045] As an improvement to the above technical solution, the method further includes a semantic fusion module for fusing numerical relationships and table structure information, including: a gating mechanism that dynamically adjusts the fusion weight of numerical information and graph structure information according to the needs of the node; adaptive fusion that obtains the final node representation through weighted summation and optimizes the information selection of the model in different reasoning steps; in the expression generation stage, the contextual information of the input and decoding history is fused through the attention mechanism, the score of each character is calculated, and the character with the highest score is selected as the prediction result.

[0046] As an improvement to the above technical solution, the expression generation step includes: using a decoder, such as a long short-term memory network (LSTM), to generate a computable expression based on the learned node representation; the decoding space for expression generation includes numerical nodes, special characters or constants, and memory characters; the contextual information of the input and decoding history is fused through the attention mechanism, the score of each character is calculated, and the character with the highest score is selected as the prediction result; during the training process, the expression generation ability of the model is optimized through cross-entropy loss, and accurate mathematical expressions or numerical answers are generated by deeply integrating the structured information of the table with the semantic information of the text, which significantly improves the accuracy, interpretability and applicability of the question-answering system, and solves the problems of table structure information loss and insufficient numerical relationship modeling in the existing technology; through heterogeneous graph modeling and relationship-aware graph neural network, the two-dimensional structure and hierarchical semantic information of the table are explicitly retained and utilized; the model is enhanced through numerical representation. Block, explicitly modeling the quantitative relationship between numerical values, and improving the model's understanding of numerical reasoning; through the semantic fusion module, dynamically integrating numerical information and structural information, and optimizing the model's information selection in different reasoning steps. The method also includes the step of training the model, and the training data includes labeled questions, texts, tables, and corresponding mathematical expressions or numerical answers. By optimizing the parameters of the heterogeneous evidence retrieval module, the heterogeneous graph construction module, the graph neural network encoding module, the numerical representation enhancement module, the semantic fusion module, and the expression generation module, the model achieves the best performance on the training data; during the training process, the performance of the text retrieval module is optimized through comparative learning; the performance of the table retrieval module is optimized through cross entropy loss and ranking loss; the performance of the graph neural network encoding module and the expression generation module are optimized through node classification loss and expression generation loss; the performance of the numerical representation enhancement module is optimized through numerical relationship recognition loss.

[0047] As an improvement to the above technical solution, the method further includes a dynamic decoding strategy based on reinforcement learning to optimize the decoding path selection during the expression generation process. Specifically, it includes: designing a reinforcement learning agent to select the optimal decoding action at each step of expression generation; dynamically adjusting the decoding strategy based on the accuracy and complexity of the generated expression through a reward mechanism; combining supervised learning and reinforcement learning during training to enable the model to learn the optimal decoding path and reduce the probability of error generation; in multi-step numerical reasoning tasks, the dynamic decoding strategy adaptively adjusts the numerical comparison and operation order in the decoding process to improve the generalization ability of the model. The method further includes an adaptive reasoning path optimization mechanism to dynamically adjust the information flow and computation path in the reasoning process to improve reasoning efficiency and accuracy. Specifically, it includes: introducing reasoning path nodes in the heterogeneous graph to represent the computation path in the reasoning process; dynamically selecting the optimal reasoning path through a path attention mechanism to reduce invalid calculations and error accumulation; adaptively adjusting the depth and breadth of the reasoning path based on the complexity of the problem and the structural characteristics of the data; and dynamically adjusting the order and method of using intermediate results in multi-step reasoning tasks through a path optimization mechanism to improve the robustness of reasoning.

[0048] Working principle and usage of the present invention:

[0049] When in use, the system first receives a document containing text and tables and a question raised by the user, and combines the information in the text and table to perform numerical reasoning, such as calculation, comparison or statistics. Through the heterogeneous evidence retrieval module, the system retrieves supporting evidence based on the question raised by the user, while retaining the two-dimensional structural integrity of the table to avoid information loss caused by serialization processing. It uses a pre-trained language model (such as RoBERTa) to encode the question and text paragraphs, calculates the semantic similarity score between them, and selects the paragraphs with the highest scores as candidate text evidence. It adopts a structured retrieval method and uses a table encoder (such as TaBERT) Encode the question and table, classify the encoding results through the multi-layer perceptron (MLP), predict the relevance between the question and the table, select the table with high relevance as the candidate table evidence, introduce the idea of contrastive learning in text retrieval, and use the negative sampling strategy within the batch to optimize the model performance; in table retrieval, output the predicted probability of the relevance between the question and the table through MLP, and based on the retrieved evidence, systematically construct a heterogeneous graph. The nodes and edges in the graph represent different semantic information and the relationship between them: use the relational perceptual graph neural network (R-GAT) to encode the heterogeneous graph, capture the interactive information between table and text data, and the layer information within the table. Secondary dependencies: Specific representation methods are designed for different node types. Initial node features are generated by aggregating subword or title information. Attention scores between nodes are calculated using a relational attention mechanism. Messages from neighboring nodes and edges are aggregated to update node representations. This prevents similar node representations caused by too many graph neural network layers. Evidence is calculated for each numerical node, and a learnable threshold is introduced to adaptively filter evidence nodes. A numerical representation enhancement module explicitly models quantitative relationships between numerical values, improving the model's understanding of numerical comparisons. Numerical values are mapped to logarithmic space and then discretized to address information imbalance caused by excessively large or small numerical ranges. Logarithmically encoded values are mapped to a fixed-dimensional space to capture relative positional information and obtain contextual semantic representations of the values. The contributions of absolute and relative semantics are adaptively adjusted based on task requirements. The magnitude relationship between any two values is measured. Numerical pair relationship recognition is treated as a three-class classification task. A cross-entropy loss and ranking loss are combined to identify the maximum and minimum values while maintaining a clear separation from other values. A semantic fusion module dynamically integrates numerical information and table structure information to optimize the model's information selection at different inference steps. The fusion weights of numerical information and graph structure information are dynamically adjusted based on node requirements.The final node representation is obtained through weighted summation, and the model's information selection in different reasoning steps is optimized. A decoder (such as a long short-term memory network (LSTM)) is used to generate computable expressions based on the learned node representation. The contextual information of the input and decoding history is integrated, the score of each character is calculated, and the character with the highest score is selected as the prediction result. The model's expression generation ability is optimized through cross-entropy loss. Combined with supervised learning and reinforcement learning, the model can learn the optimal decoding path and reduce the probability of error generation. A reinforcement learning agent is designed to select the optimal decoding action at each step of expression generation. The decoding strategy is dynamically adjusted according to the accuracy and complexity of the generated expression. Reasoning path nodes are introduced in heterogeneous graphs. The optimal reasoning path is dynamically selected through the path attention mechanism to reduce invalid calculations and error accumulation. The depth and breadth of the reasoning path are adaptively adjusted according to the complexity of the problem and the structural characteristics of the data. The table-text numerical reasoning question answering method based on joint modeling can deeply integrate the structured information of the table with the semantic information of the text to generate precise mathematical expressions or numerical answers, significantly improving the accuracy, interpretability, and applicability of the question answering system.

[0050] Example: Question Answering of Numerical Reasoning Based on Tables and Texts in Financial Data Analysis

[0051] The user is analyzing a company's financial statements. The statements are presented in a table format, along with a text describing the company's financial status. The user wants to know "what was the company's average sales in the past three years in which its net profit exceeded 1 million yuan."

[0052] Input data: Description of the company's financial performance over the past three years, for example: "The company achieved significant sales growth in 2022, with a net profit of 1.2 million yuan."

[0053] The table is as follows:

[0054] years Sales (10,000 yuan) Net profit (10,000 yuan) 2022 500 120 2023 450 90 2024 600 110

[0055] Receive input: Receive a document containing the above text and table.

[0056] Receive a question from a user: "What was the company's average sales in the past three years in which its net profit exceeded 1 million yuan?"

[0057] Text Data Retrieval: Use a pre-trained language model (such as RoBERTa) to encode the question and text paragraphs, calculate semantic similarity, and select the text paragraph most relevant to the question as candidate text evidence. For example, if the similarity score between the text paragraph and the question is 0.85, it will be selected as candidate text evidence.

[0058] Tabular Data Retrieval: Using a structured retrieval approach, we use a table encoder (such as TaBERT) to encode the question and the table. We then use a multi-layer perceptron (MLP) to classify the encoded results and predict the relevance between the question and the table. Assuming the MLP output predicts a relevance probability of 0.9, we consider the table as candidate table evidence.

[0059] Construct a heterogeneous graph, including the following nodes and edges: question node: represents the semantic information of the question raised by the user; text paragraph node: represents the semantic information of the candidate text evidence; value node: represents all numbers appearing in the question, text and table, such as 120, 500, 450, 90, 600, 110; row header node and column header node: represent the row and column structure information of the table, such as "year", "sales", "net profit"; hierarchy header node: represents the nested hierarchy information in the table (if any); date node: represents the time information appearing in the question, text and table, such as 2022, 2023, 2024; table node: represents the global information of the entire table; special character or constant node: represents special characters or constants used for expression generation, such as "+", "-", " / ", etc.

[0060] Edge types are defined, such as edges from row headers to values, from column headers to values, from questions to text paragraphs, and from questions to tables. A relation-aware graph neural network (R-GAT) is used to encode heterogeneous graphs. Specific representation methods are designed for different node types. Initial node features are generated by aggregating subword or title information. Attention scores between nodes are calculated using a relational attention mechanism. Node representations are updated by aggregating messages from neighboring nodes and edges. Layer normalization and residual connections are used to prevent similar node representations caused by too many graph neural network layers. For the node classification task, each numerical node is evaluated as evidence, and a learnable threshold is introduced to adaptively filter out evidence nodes. For example, the numerical nodes relevant to the question are filtered out as 120, 500, 600, and 110.

[0061] The values 500, 450, 600, 120, 90, and 110 are mapped to logarithmic space and then discretized to address information imbalance caused by excessively large or small numerical ranges. The logarithmically encoded values are then mapped to a fixed-dimensional space to capture the relative position of the values. The contributions of absolute and relative semantics are adaptively adjusted based on task requirements, measuring the magnitude relationship between any two values. The recognition of value pair relationships is treated as a three-class classification task. For example, 120 > 100, 110 > 100, and 90 < 100 are identified. A cross-entropy loss and a ranking loss are combined to identify the maximum and minimum values, maintaining a clear separation from other values. For example, 600 is identified as the maximum sales value and 90 as the minimum net profit value. The fusion weights of numerical and graph structural information are dynamically adjusted based on node requirements. The final node representation is obtained through weighted summation, optimizing the model's information selection at different inference steps.

[0062] A decoder (such as a long short-term memory (LSTM) network) generates computable expressions based on learned node representations. The decoding space includes numeric nodes (500, 600), special characters or constants ("+", " / "), and memory characters (such as "#1" representing intermediate results). Contextual information from the input and decoding history is integrated to calculate a score for each character and select the highest-scoring character as the prediction. For example, the expression "(500+600) / 2" is generated. The model's expression generation capability is optimized using cross-entropy loss. A reinforcement learning agent is designed to select the optimal decoding action at each step of expression generation. The decoding strategy is dynamically adjusted based on the accuracy and complexity of the generated expression. Inference path nodes are introduced into the heterogeneous graph. The path attention mechanism dynamically selects the optimal inference path, reducing inefficient computation and error accumulation. For example, the depth and breadth of the inference path are adaptively adjusted based on the complexity of the problem and the structural characteristics of the data.

[0063] Finally, the system generates and calculates the expression "(500+600) / 2" and comes up with the answer: the average sales of the company in the years with net profit exceeding 1 million yuan in the past three years is 5.5 million yuan.

[0064] The above description is only used to illustrate the technical solution of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, it should be understood by those skilled in the art that the technical solutions described in the aforementioned embodiments can still be modified, or some or all of the technical features therein can be replaced by equivalents. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied to other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A table-text numerical reasoning question-answering method based on joint modeling, characterized by: The following steps are involved: Step 1: Receive a document containing text and tables and questions raised by the user; Step 2: Using the heterogeneous evidence retrieval module, support evidence is retrieved based on the question, preserving the two-dimensional structural integrity of the table and avoiding information loss caused by serialization; Step 3: Based on the retrieved evidence, a heterogeneous graph is constructed. The heterogeneous graph includes question nodes, text paragraph nodes, table cell nodes, and table structure information nodes. The edges between the nodes represent the semantic associations between them. Step 4: Encode the heterogeneous graph using a relation-aware graph neural network to capture the interaction information between table and text data as well as the hierarchical dependencies within the table; Step 5: Generate mathematical expressions or numerical answers to answer questions based on the learned node representations.

2. The table-text numerical reasoning question-answering method based on joint modeling according to claim 1 is characterized by: The heterogeneous evidence retrieval module includes the following steps: for text data, using a dense retrieval method, using a pre-trained language model to encode questions and text paragraphs, calculating similarity scores, and selecting several paragraphs with the highest scores as candidate text evidence; for tabular data, using a structured retrieval method, using a table encoder to encode questions and tables, using a classifier to predict the relevance between questions and tables, and selecting tables with high relevance as candidate tabular evidence; introducing the idea of contrastive learning in text retrieval, and using a batch negative sampling strategy to optimize model performance; in table retrieval, using a multi-layer perceptron (MLP) to classify the output of the table encoder and output the predicted probability of the relevance between the question and the table.

3. The table-text numerical reasoning question-answering method based on joint modeling according to claim 2 is characterized by: The construction of the heterogeneous graph includes the following node types: question nodes, which represent the semantic information of the questions raised by the user; text paragraph nodes, which represent the semantic information of the retrieved text evidence; value nodes, which represent all numbers appearing in the questions, texts and tables; row header nodes and column header nodes, which represent the row and column structure information of the table; hierarchical header nodes, which represent the nested hierarchical structure information in the table; date nodes, which represent the time information appearing in the questions, texts and tables; table nodes, which represent the global information of the entire table; special character or constant nodes, which represent special characters or constants used for expression generation.

4. The table-text numerical reasoning question-answering method based on joint modeling according to claim 3 is characterized by: The construction of the heterogeneous graph includes the following types: internal relationship edges in a table, from column header to value, from table to row header; internal relationship edges in a paragraph or question, from question to document, from question to table; relationship edges between paragraph and table, relationship edges between tables, and cross-modal relationship edges.

5. The table-text numerical reasoning question-answering method based on joint modeling according to claim 4 is characterized by: The relationship-aware graph neural network includes: node initialization, designing specific representation methods according to different node types, generating initial features of nodes by aggregating subword information or title information; message passing mechanism, calculating attention scores between nodes through the relational attention mechanism, aggregating messages of neighboring nodes and edges to update node representations; layer normalization and residual connection, preventing the problem of similar node representations caused by too many layers of the graph neural network; node classification task, calculating the evidence for each numerical node, and introducing a learnable threshold to adaptively screen evidence nodes.

6. The table-text numerical reasoning question-answering method based on joint modeling according to claim 5 is characterized by: The method further includes a numerical representation enhancement module for explicitly modeling the quantitative relationship between numerical values, including: multi-granularity semantic encoding, which captures the magnitude information and contextual relationship of numerical values through absolute semantic channels and relative semantic channels respectively; a numerical relationship recognition loss function, which optimizes the size relationship and extreme value relationship between numerical values to improve the model's understanding of numerical comparison; a binary relationship loss, which measures the size relationship between any two numerical values and regards the recognition of numerical pair relationships as a three-classification task; and an extreme value recognition loss, which combines cross-entropy loss and ranking loss to identify the maximum and minimum values and maintain a clear interval between them and other numerical values.

7. The table-text numerical reasoning question-answering method based on joint modeling according to claim 6 is characterized by: The multi-granularity semantic encoding includes: logarithmic bucket encoding, mapping the numerical value to the logarithmic space and then discretizing it, mapping the logarithmically encoded numerical value to a fixed-dimensional space to capture the relative position information of the numerical value; subword attention pooling, obtaining the contextual semantic representation of the numerical value; and a gated fusion mechanism, adaptively adjusting the contribution of absolute semantics and relative semantics according to task requirements.

8. The table-text numerical reasoning question-answering method based on joint modeling according to claim 7 is characterized by: The method further includes a semantic fusion module for fusing numerical relationships and table structure information, including: a gating mechanism that dynamically adjusts the fusion weight of numerical information and graph structure information according to node requirements; adaptive fusion that obtains the final node representation through weighted summation and optimizes the model's information selection in different reasoning steps; in the expression generation stage, the contextual information of the input and decoding history is fused through the attention mechanism, the score of each character is calculated, and the character with the highest score is selected as the prediction result.

9. The table-text numerical reasoning question-answering method based on joint modeling according to claim 8 is characterized by: The expression generation step includes: using a decoder to generate a computable expression based on the learned node representation; the decoding space for expression generation includes numerical nodes, special characters or constants, and memory characters; fusing the contextual information of the input and decoding history through the attention mechanism, calculating the score of each character and selecting the character with the highest score as the prediction result; during the training process, optimizing the expression generation ability of the model through cross entropy loss, generating accurate mathematical expressions or numerical answers through deep integration of the structured information of the table and the semantic information of the text, explicitly retaining and utilizing the two-dimensional structure and hierarchical semantic information of the table through heterogeneous graph modeling and relationship-aware graph neural network; explicitly modeling the quantitative relationship between numerical values through the numerical representation enhancement module, and improving the model's understanding of numerical reasoning; through the semantic fusion module, Dynamically integrate numerical information and structural information to optimize the information selection of the model in different reasoning steps. The method also includes a step of training the model. The training data includes labeled questions, texts, tables, and corresponding mathematical expressions or numerical answers. By optimizing the parameters of the heterogeneous evidence retrieval module, heterogeneous graph construction module, graph neural network encoding module, numerical representation enhancement module, semantic fusion module, and expression generation module, the model achieves optimal performance on the training data. During the training process, the performance of the text retrieval module is optimized through comparative learning; the performance of the table retrieval module is optimized through cross entropy loss and ranking loss; the performance of the graph neural network encoding module and the expression generation module are optimized through node classification loss and expression generation loss; and the performance of the numerical representation enhancement module is optimized through numerical relationship recognition loss.

10. The table-text numerical reasoning question-answering method based on joint modeling according to claim 9 is characterized in that: The method further includes a dynamic decoding strategy based on reinforcement learning, which is used to optimize the decoding path selection during the expression generation process, specifically including: designing a reinforcement learning agent to select the optimal decoding action at each step of expression generation; dynamically adjusting the decoding strategy according to the accuracy and complexity of the generated expression through a reward mechanism; during the training process, combining supervised learning and reinforcement learning to enable the model to learn the optimal decoding path and reduce the probability of error generation; in multi-step numerical reasoning tasks, adaptively adjusting the numerical comparison and operation order in the decoding process through a dynamic decoding strategy to improve the generalization ability of the model. The method further includes an adaptive reasoning path optimization mechanism for dynamically adjusting the information flow and calculation path in the reasoning process to improve reasoning efficiency and accuracy, specifically including: introducing reasoning path nodes in a heterogeneous graph to represent the calculation path in the reasoning process; dynamically selecting the optimal reasoning path through a path attention mechanism to reduce invalid calculations and error accumulation; adaptively adjusting the depth and breadth of the reasoning path according to the complexity of the problem and the structural characteristics of the data; in multi-step reasoning tasks, dynamically adjusting the order and method of using intermediate results through a path optimization mechanism to improve the robustness of reasoning.

Images